Unifying Remote Sensing Image Retrieval and Classification with Robust Fine-tuning

Dimitri Gominski, Valérie Gouet-Brunet, Liming ChenarXiv 2021 · OpenReview

TL;DR

This paper tackles the challenge of limited annotated data in remote sensing by introducing SF300, a large-scale dataset for image retrieval and classification. They also propose an adversarial fine-tuning method that improves the robustness of image descriptors, leading to better performance across multiple datasets.

What Problem Does This Paper Address?

Analyzing high-resolution remote sensing images is crucial for many applications, but it's hindered by a lack of sufficient annotated data for training deep learning models. This has led to a proliferation of small, dataset-specific methods that are hard to compare and generalize. Furthermore, existing benchmarks for remote sensing image retrieval (RSIR) and classification (RSC) are often small and don't adequately represent the real-world variations in aerial imagery. The authors argue that RSC can be seen as a special case of RSIR, and that RSIR should be the primary focus for advancing remote sensing image understanding. They aim to unify these tasks and establish a more robust evaluation framework.

Methods

The core of the paper's contribution lies in two main areas:

  1. SF300 Dataset: A new, large-scale dataset is introduced, comprising over 300,000 aerial images (both vertical and oblique) with 27,502 unique classes. This dataset is designed to be a more comprehensive benchmark for RSIR and RSC, featuring geolocations as classes rather than high-level semantics, which introduces more realistic intra-class variability and inter-class similarity.
  2. Adversarial Fine-tuning Framework: The authors propose a fine-tuning method that enhances the robustness of global image descriptors. This method builds upon a standard fine-tuning approach using the GeM pooling layer and a whitening network. The key innovation is the addition of an "orientation classifier" that is trained adversarially. This classifier learns to predict the orientation of an image (e.g., vertical, oblique at 45 degrees). By jointly optimizing the feature extractor and the orientation classifier, the framework encourages the main descriptor to be invariant to orientation variations, thereby improving its generalization capabilities. The objective function balances the classification loss with the orientation prediction loss, controlled by a weighting parameter $\alpha$.

Datasets

  • SF300: A newly introduced large-scale dataset for training and testing RSIR and RSC. It contains 308,353 images in the training set and 21,844 in the test set, with 27,502 and 2,421 classes respectively. Images are 512x512 pixels and include both vertical and oblique aerial views.
  • Nine other remote sensing datasets: AID, BCS, PatternNet, RESISC45, RSI-CB, RSSCN7, SIRI-WHU, UCM, and WHU-RS19 were used for evaluation to demonstrate the generalizability of their proposed framework.

Key Insights and Ideas

  • Unifying RSIR and RSC: The paper posits that RSIR is a more fundamental task for remote sensing image comprehension, as RSC can be viewed as a subset of RSIR. This perspective helps in developing more generalizable methods.
  • The Importance of Large-Scale, Diverse Data: The SF300 dataset highlights the need for larger, more representative datasets to overcome the limitations of existing small benchmarks and to enable more robust fine-tuning of deep learning models.
  • Orientation as a Source of Variability: Variations in image orientation (due to camera angle) are identified as a significant source of variability that negatively impacts descriptor robustness. The adversarial approach directly addresses this.
  • Adversarial Training for Robustness: Using an adversarial classifier to predict image orientation proves effective in forcing the main feature extractor to learn orientation-invariant representations, leading to improved retrieval performance.
  • Dataset-Specific Performance Fluctuations: The authors acknowledge that their adversarial fine-tuning method doesn't improve performance on all datasets, suggesting that dataset characteristics (spatial resolution, inter-class similarity, etc.) play a significant role. This underscores the need for broad evaluation.

Key Results

  • The SF300 dataset offers a significant size improvement (10x) over the largest existing dataset (RESISC45).
  • The proposed fine-tuning framework, when applied to SF300, consistently improves retrieval and classification performance across nine different datasets compared to an ImageNet-pretrained baseline.
  • The adversarial fine-tuning (AdvFT) method shows improvements on seven out of the nine evaluated datasets, with mAP gains ranging from less than 1% to 5%.
  • For the SF300 dataset itself, the fine-tuned models (FT and AdvFT) achieve significantly higher performance (e.g., ~58% mAP for FT, ~59.5% mAP for AdvFT) compared to the ImageNet pretrained baseline (~48% mAP).
  • The AdvFT model achieves the highest mean mAP across all tested datasets (59.48%) and the second-highest mean OA/mP@1 (91.79%), closely following the FT model (91.62%).

Limitations and Future Work

  • Performance Variability: The adversarial fine-tuning method did not improve performance on all datasets, which the authors attribute to varying dataset characteristics. This suggests that further research might be needed to tailor the adversarial approach to specific dataset properties.
  • Class Definition in SF300: The unique class definition in SF300 (geolocations) leads to high inter-class similarity, making it a more challenging dataset but potentially limiting direct comparison with datasets using semantic classes.
  • Future Work: The authors suggest that their framework can be adapted to other state-of-the-art global descriptors. Further exploration into the reasons behind performance variations across datasets and potential improvements to the adversarial training strategy could be valuable.

Summary updated

Abstract

Advances in high resolution remote sensing image analysis are currently hampered by the difficulty of gathering enough annotated data for training deep learning methods, giving rise to a variety of small datasets and associated dataset-specific methods. Moreover, typical tasks such as classification and retrieval lack a systematic evaluation on standard benchmarks and training datasets, which make it hard to identify durable and generalizable scientific contributions. We aim at unifying remote sensing image retrieval and classification with a new large-scale training and testing dataset, SF300, including both vertical and oblique aerial images and made available to the research community, and an associated fine-tuning method. We additionally propose a new adversarial fine-tuning method for global descriptors. We show that our framework systematically achieves a boost of retrieval and classification performance on nine different datasets compared to an ImageNet pretrained baseline, with currently no other method to compare to.
BibTeX
@misc{gominski2021unifying,
  author = {Dimitri Gominski and Valérie Gouet-Brunet and Liming Chen},
  title = {{Unifying Remote Sensing Image Retrieval and Classification with Robust Fine-tuning}},
  year = {2021},
  eprint = {2102.13392},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.13392},
}

Loading related content...

Figure