Rotation Invariant Aerial Image Retrieval with Group Convolutional Metric Learning

Hyunseung Chung, Woo-Jeoung Nam, Seong-Whan LeearXiv 2020 · OpenReview

TL;DR

This paper introduces a novel method for retrieving aerial images that is robust to rotational variations, a common challenge in remote sensing. By combining group convolutions, channel attention, and triplet loss, their method significantly outperforms existing techniques in both standard and rotated image retrieval scenarios.

What Problem Does This Paper Address?

Remote sensing image retrieval (RSIR) aims to find similar images in a large database given a query image. However, aerial images, captured by satellites or aircraft, suffer from significant variations in shooting range, angle, and location. This diversity, particularly rotational differences, makes it difficult for traditional image retrieval methods to accurately identify matching images. Existing Convolutional Neural Networks (CNNs) are often trained on upright images and struggle with these arbitrary orientations. The paper addresses this gap by developing a retrieval system that can effectively handle these rotational variations, ensuring more reliable retrieval of identical locations despite different viewing angles.

Methods

The proposed method, "Attentive Group Convolutional Metric Learning Network," employs a two-step training strategy:

  1. Classification Pre-training: A Group Convolutional Neural Network (G-CNN) is first trained on the Aerial Image Dataset (AID) for classification. G-CNNs are designed to be inherently rotation-equivariant by using rotated filters, allowing them to learn features that are consistent across different orientations without explicitly rotating the input images. This step helps the network learn general features of aerial imagery.
  2. Metric Learning Fine-tuning: The pre-trained G-CNN is then fine-tuned for retrieval using a metric learning approach. This involves:
  3. Channel Attention: An attention module is integrated into each group convolution stage. This module helps the network focus on the most salient features within the feature maps, further refining the representations.
  4. Triplet Loss: This loss function is used to train the network to learn discriminative embeddings. It works by ensuring that the distance between an anchor image and a positive (similar) image is smaller than the distance between the anchor and a negative (dissimilar) image, with a specified margin.

The authors utilize a ResNet-34 backbone, replacing standard convolutional layers with group convolutional layers and adapting batch normalization to handle the increased channel dimensions.

Datasets

  • Aerial Image Dataset (AID): Used for the initial classification pre-training. It contains 30 classes and 10,000 images, known for its diverse rotational images.
  • Google Earth South Korea Dataset: A multi-temporal dataset with 40,000 satellite images (1080x1080 pixels) of South Korea. Used for fine-tuning and retrieval evaluation. The dataset's multi-temporal nature allows for creating "positive" samples that represent the same location but at different times.
  • NWPU-RESISC45 Dataset: A large-scale aerial image dataset with 45 classes and 31,500 images (256x256 pixels). Used for fine-tuning and retrieval evaluation, offering high within-class diversity.

The authors also created rotated versions of the test sets for Google Earth South Korea and NWPU-RESISC45 to specifically evaluate the model's performance under rotational variations.

Key Insights and Ideas

  • Group Convolutions for Rotation Invariance: The core idea is that by rotating filters instead of images, G-CNNs can learn features that are invariant to rotation, significantly reducing computational overhead compared to methods that rely on data augmentation with many rotations.
  • Channel Attention Enhances Salient Features: Integrating channel attention within the G-CNN framework helps the model to selectively emphasize important feature channels, leading to more robust representations, especially in the presence of complex spatial transformations.
  • Two-Step Training is Crucial: The paper highlights the importance of a two-step training process: first, learning general aerial image features via classification on a diverse dataset (AID), and then fine-tuning for the specific retrieval task using metric learning on target datasets. This approach helps bridge the domain gap between general datasets like ImageNet and specialized remote sensing data.
  • Triplet Loss for Discriminative Embeddings: Triplet loss is effective in learning embeddings where similar images are clustered together and dissimilar images are pushed apart, which is fundamental for accurate image retrieval.

Key Results

  • Superior Performance: The proposed attentive G-CNN with triplet loss consistently outperformed state-of-the-art methods on both the Google Earth South Korea and NWPU-RESISC45 datasets, achieving higher Recall@n scores across various values of 'n'.
  • Robustness to Rotation: The model demonstrated significantly better performance on rotated datasets compared to baseline methods, indicating its strong rotational invariance. For instance, on the rotated Google Earth South Korea dataset, the proposed method achieved Recall@100 of 86.3%, while other methods struggled to reach 70%.
  • Visualization of Feature Importance: Class Activation Maps (CAMs) revealed that the proposed method's activated regions were more stable and expressive across different rotations compared to standard CNNs, visually confirming its enhanced adaptability.

Limitations and Future Work

  • Computational Cost: While group convolutions reduce the need for image rotation, they can increase the number of feature maps, potentially leading to a higher parameter count. The authors addressed this by halving filters, but it's an ongoing consideration.
  • Triplet Mining: The effectiveness of triplet loss relies on selecting informative triplets. The paper mentions using a dense triplet mining method, but further exploration into more advanced triplet selection strategies could be beneficial.
  • Future Work: The authors suggest that further research could explore more advanced attention mechanisms and group convolution types to further enhance retrieval performance. Investigating the application of this method to other types of remote sensing data or tasks could also be a promising direction.

Summary updated

Abstract

Remote sensing image retrieval (RSIR) is the process of ranking database images depending on the degree of similarity compared to the query image. As the complexity of RSIR increases due to the diversity in shooting range, angle, and location of remote sensors, there is an increasing demand for methods to address these issues and improve retrieval performance. In this work, we introduce a novel method for retrieving aerial images by merging group convolution with attention mechanism and metric learning, resulting in robustness to rotational variations. For refinement and emphasis on important features, we applied channel attention in each group convolution stage. By utilizing the characteristics of group convolution and channel-wise attention, it is possible to acknowledge the equality among rotated but identically located images. The training procedure has two main steps: (i) training the network with Aerial Image Dataset (AID) for classification, (ii) fine-tuning the network with triplet-loss for retrieval with Google Earth South Korea and NWPU-RESISC45 datasets. Results show that the proposed method performance exceeds other state-of-the-art retrieval methods in both rotated and original environments. Furthermore, we utilize class activation maps (CAM) to visualize the distinct difference of main features between our method and baseline, resulting in better adaptability in rotated environments.
BibTeX
@misc{chung2020rotation,
  author = {Hyunseung Chung and Woo-Jeoung Nam and Seong-Whan Lee},
  title = {{Rotation Invariant Aerial Image Retrieval with Group Convolutional Metric Learning}},
  year = {2020},
  eprint = {2010.09202},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.09202},
}
Figure