Multi-Granularity Canonical Appearance Pooling for Remote Sensing Scene Classification

Shidong Wang, Yu Guan, Ling ShaoIEEE TIP 2020 · DOI · OpenReview

TL;DR

This paper introduces Multi-Granularity Canonical Appearance Pooling (MG-CAP), a novel framework for remote sensing scene classification that tackles the challenge of visual-semantic discrepancies. By learning features from progressively cropped image regions and transforming them into Gaussian covariance matrices, MG-CAP effectively captures latent ontological structures and achieves state-of-the-art results.

What Problem Does This Paper Address?

Remote sensing scene classification (RSSC) is crucial for applications like land use mapping and urban planning. However, it faces two major challenges: large visual-semantic discrepancies and significant intra-class variations coupled with inter-class similarities. The former arises because high-level semantic labels are difficult to align with pixel-level features without detailed annotations, which are labor-intensive and subjective to obtain. The latter means that images within the same category can look very different (e.g., different types of railways), while images from different categories can appear similar (e.g., freeways and railways). Existing methods struggle to bridge this gap, especially when dealing with the hierarchical nature of remote sensing ontologies.

Methods

The core of the proposed MG-CAP framework is its multi-granularity approach. It progressively crops the input image into different "grains" (resolutions), from coarse to fine. For each grain, a Siamese-style network is used to extract features from transformed versions of the cropped image. These features are then converted into Gaussian covariance matrices, which are known to capture second-order statistics and preserve spatial structures.

A key innovation is the "Canonical Appearance Pooling" layer. This layer uses a maxout operation to select the most representative covariance matrix from the transformed instances within a specific grain, effectively learning a "canonical appearance" that is invariant to transformations. These canonical covariance matrices from different granularities are then averaged to form a single, robust representation.

To further enhance discriminative power, the framework employs eigenvalue decomposition (EIG) with specialized matrix normalizations. Crucially, the authors provide a stable, GPU-accelerated EIG-decomposition function and demonstrate its back-propagation, addressing a significant computational bottleneck in previous work.

Datasets

The paper evaluates its framework on three widely used remote sensing scene classification datasets:

  • NWPU-RESISC45: Contains 31,500 images across 45 classes.
  • AID (Aerial Image Dataset): Features 10,000 images in 30 classes.
  • UC Merced: A smaller dataset with 2,100 images across 21 classes.

Key Insights and Ideas

  • Latent Ontologies: The paper posits that remote sensing datasets possess a latent hierarchical structure (ontology) that can be exploited to reduce visual-semantic discrepancies. MG-CAP aims to automatically learn this structure.
  • Canonical Appearance: By transforming image patches and selecting the "best" representation via a maxout operation, the model learns a canonical view that is robust to variations and highlights essential discriminative features.
  • Covariance Matrices as Features: Using Gaussian covariance matrices as features is a powerful way to capture second-order statistics and spatial relationships, outperforming simple feature concatenation.
  • GPU-Accelerated EIG-Decomposition: Making EIG-decomposition efficient on GPUs is a significant practical advancement, enabling the use of complex matrix operations in deep learning pipelines.
  • Granularity Matters: The experiments show that combining multiple granularities (different crop sizes) is crucial for optimal performance, but over-cropping can be detrimental.

Key Results

  • State-of-the-Art Performance: MG-CAP, particularly the Sqrt-E variant, achieves state-of-the-art accuracy on the NWPU-RESISC45 dataset, reaching 92.95% with a 20% training ratio, significantly outperforming previous methods.
  • Competitive Results on Other Datasets: The framework also demonstrates strong performance on the AID and UC Merced datasets, often achieving the best results among compared methods.
  • Efficiency: Despite its complexity, the GPU-based MG-CAP achieves competitive inference times (0.216 sec/img) compared to other advanced methods like RTN.
  • Ablation Studies: Granularity analysis shows that combining three granularities yields the best results, with the second granularity often being the most informative. The number of transformations also positively impacts accuracy, up to a point.

Limitations and Future Work

  • Memory Burden: While GPU acceleration helps, the matrix operations and multiple streams can still lead to a significant memory footprint.
  • Inference Time: Although competitive, the inference time is still higher than some simpler methods, and further optimization is desired.
  • Future Work: The authors suggest investigating ways to compress model parameters to reduce memory usage while maintaining classification performance. They also aim to explore more sophisticated matrix normalization techniques and potentially more advanced deep learning architectures.

Summary updated

Abstract

Recognising remote sensing scene images remains challenging due to large visual-semantic discrepancies. These mainly arise due to the lack of detailed annotations that can be employed to align pixel-level representations with high-level semantic labels. As the tagging process is labour-intensive and subjective, we hereby propose a novel Multi-Granularity Canonical Appearance Pooling (MG-CAP) to automatically capture the latent ontological structure of remote sensing datasets. We design a granular framework that allows progressively cropping the input image to learn multi-grained features. For each specific granularity, we discover the canonical appearance from a set of pre-defined transformations and learn the corresponding CNN features through a maxout-based Siamese style architecture. Then, we replace the standard CNN features with Gaussian covariance matrices and adopt the proper matrix normalisations for improving the discriminative power of features. Besides, we provide a stable solution for training the eigenvalue-decomposition function (EIG) in a GPU and demonstrate the corresponding back-propagation using matrix calculus. Extensive experiments have shown that our framework can achieve promising results in public remote sensing scene datasets.
BibTeX
@article{wang2020multi,
  author = {Shidong Wang and Yu Guan and Ling Shao},
  title = {{Multi-Granularity Canonical Appearance Pooling for Remote Sensing Scene Classification}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  doi = {10.1109/tip.2020.2983560},
}
Figure