Relation-Attention Networks for Remote Sensing Scene Classification

Xin Wang, Lin Duan, Chen Ning, Huiyu ZhouIEEE JSTARS 2021 · DOI · OpenReview

Abstract

Remote sensing (RS) scene classification plays an important role in a wide range of RS applications. Recently, convolutional neural networks (CNNs) have been applied to the field of scene classification in RS images and achieved impressive performance. However, to classify RS scenes, most of the existing CNN methods either utilize the high-level features from the last convolutional layer of CNNs, missing much important information existing at the other levels, or directly fuse the features at different levels, bringing redundant and/or mutually exclusive information. Inspired by the attention mechanism of the human visual system, in this article, we explore a novel relation-attention model and design an end-to-end relation-attention network (RANet) to learn powerful feature representations of multiple levels to further improve the classification performance. First, we propose to extract convolutional features at different levels by pretrained CNNs. Second, a multiscale feature computation module is constructed to connect features at different levels and generate multiscale semantic features. Third, a novel relation-attention model is designed to focus on the critical features whilst avoiding the use of redundant and even distractive ones by exploiting the scale contextual information. Finally, the resulting relation-attention features are concatenated and fed into a softmax layer for the final classification. Experiments on four well-known RS scene classification datasets (UC-Merced, WHU-RS19, AID, and OPTIMAL-31) show that our method outperforms some state-of-the-art algorithms.

Introduction

Classifying scenes in remote sensing—identifying whether an image depicts an airport, a forest, or a residential area—is notoriously difficult. Unlike standard photos, satellite images contain objects at vastly different scales, orientations, and complex spatial layouts. Standard Convolutional Neural Networks (CNNs) usually struggle with two primary issues in this field: information loss and redundancy. Most models rely solely on high-level features from the final layer, missing fine-grained details. Conversely, when models attempt to combine multiple layers, they often become overwhelmed by "distractive" information. For instance, in a "river" scene, the water is the defining feature, but surrounding trees or cars can act as redundant data that confuses a standard classifier.

The RANet Architecture

To address these challenges, the researchers developed the Relation-Attention Network (RANet), an end-to-end framework consisting of four primary modules:

  1. Deep Feature Extraction (DFE): The system uses a pre-trained ResNet50 as a backbone to extract features at four different hierarchical levels.
  2. Multiscale Feature Computation (MSFC): Borrowing from Feature Pyramid Networks (FPN), this module creates a "top-down" pathway. It takes the strong semantic information from deep layers and passes it back to shallower layers to help them understand the broader context.
  3. Relation-Attention Feature Construction and Fusion (RAFCF): This is the core innovation. Instead of just stacking features, the model uses "Relation-Attention." It calculates attention weights by looking at the relationship between adjacent scales. It uses a 1D convolution to capture dependencies between channels efficiently, allowing the features at larger scales to "supervise" and refine the features at smaller scales.
  4. Scene Classification (SC): Finally, these refined, attention-weighted features are concatenated and fed into a softmax layer for final categorization.
The proposed RANet architecture showing the flow from Deep Feature Extraction to Scene Classification.

Experimental Evaluation

The model was rigorously tested on four benchmark datasets: UC-Merced, WHU-RS19, the large-scale Aerial Image Dataset (AID), and OPTIMAL-31. RANet consistently outperformed state-of-the-art (SOTA) methods across all four:

  • UC-Merced: Achieved an impressive 99.27% accuracy.
  • WHU-RS19: Reached 98.97% accuracy, showing superior performance even when training data was reduced to 40%.
  • AID: Obtained competitive results (95.31%), proving it can handle large-scale, diverse data.
  • Small Data Performance: On the OPTIMAL-31 dataset, RANet improved accuracy by nearly 14% compared to a fine-tuned AlexNet, proving its "Relation-Attention" is highly effective even when samples are scarce.

Visual Interpretability and Insights

A key insight of this research is the use of higher-level supervision. By letting deep layers (the "what") tell the shallow layers (the "where") what to ignore, the model becomes much more robust. Using Grad-CAM (Gradient-weighted Class Activation Mapping), the authors demonstrated that RANet focuses on the correct semantic regions. While a standard model might get distracted by the trees surrounding a river, RANet focuses squarely on the water itself.

Grad-CAM visualizations comparing incorrect predictions (distracted by background elements) vs. correct predictions by RANet.

Conclusion

RANet demonstrates that a relation-attention mechanism can significantly enhance the discriminative power of CNNs for remote sensing. By filtering out redundant background noise and focusing on critical features across multiple scales, the model achieves state-of-the-art performance. Future work will involve integrating this mechanism into Transformer-based frameworks and expanding its application to tasks like object detection and environmental monitoring.

Summary updated

Abstract

Remote sensing (RS) scene classification plays an important role in a wide range of RS applications. Recently, convolutional neural networks (CNNs) have been applied to the field of scene classification in RS images and achieved impressive performance. However, to classify RS scenes, most of the existing CNN methods either utilize the high-level features from the last convolutional layer of CNNs, missing much important information existing at the other levels, or directly fuse the features at different levels, bringing redundant and/or mutually exclusive information. Inspired by the attention mechanism of the human visual system, in this article, we explore a novel relation-attention model and design an end-to-end relation-attention network (RANet) to learn powerful feature representations of multiple levels to further improve the classification performance. First, we propose to extract convolutional features at different levels by pretrained CNNs. Second, a multiscale feature computation module is constructed to connect features at different levels and generate multiscale semantic features. Third, a novel relation-attention model is designed to focus on the critical features whilst avoiding the use of redundant and even distractive ones by exploiting the scale contextual information. Finally, the resulting relation-attention features are concatenated and fed into a softmax layer for the final classification. Experiments on four well-known RS scene classification datasets (UC-Merced, WHU-RS19, AID, and OPTIMAL-31) show that our method outperforms some state-of-the-art algorithms.
BibTeX
@article{wang2021relation,
  author = {Xin Wang and Lin Duan and Chen Ning and Huiyu Zhou},
  title = {{Relation-Attention Networks for Remote Sensing Scene Classification}},
  journal = {IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing},
  year = {2021},
  doi = {10.1109/jstars.2021.3135566},
}
Figure