Multisource Region Attention Network for Fine-Grained Object Recognition in Remote Sensing Imagery

Gencer Sumbul, Ramazan Gokberk Cinbis, Selim Aksoy

Abstract

Fine-grained object recognition in remote sensing aims to identify specific sub-categories, such as tree species, within broader classes. This task is challenging due to low between-class variance and the inherent misalignment of multisource data (e.g., RGB, multispectral, and LiDAR) at the pixel level. We present the Multisource Region Attention Network (MRAN), a unified framework that simultaneously learns data alignment and classification. By employing a region attention mechanism, MRAN assigns scores to candidate patches across different sensors to create a fused, aligned representation. Experiments on a street tree dataset demonstrate that MRAN achieves significant accuracy improvements—up to 14.3%—over traditional feature concatenation methods.

The Challenge of Fine-Grained Recognition

Traditional remote sensing classification often distinguishes between broad land-cover types like "water" or "forest." Fine-grained recognition pushes these boundaries by attempting to identify dozens of closely related species or object types. This task faces two primary hurdles:

  1. Visual Similarity: Sub-categories often share nearly identical spatial and spectral characteristics.
  2. Data Misalignment: While combining RGB, multispectral (MS), and LiDAR data provides a richer feature set, these sensors are rarely perfectly co-registered. Even a shift of a few pixels can cause a model to fuse features from a tree in one source with a sidewalk in another, leading to classification errors.

The MRAN Framework

The proposed Multisource Region Attention Network (MRAN) addresses misalignment by treating it as a learning problem rather than a preprocessing step. The architecture consists of a reference branch and multiple auxiliary branches.

Overview of the Multisource Region Attention Network (MRAN) architecture, showing how the reference RGB source guides attention across candidate regions in MS and LiDAR data.

Region Attention Mechanism

In the MRAN model, one source (typically high-resolution RGB) serves as the reference. The model extracts features from a verified location in this reference. For auxiliary sources like MS and LiDAR, the model samples a grid of region proposals around the expected location.

The core innovation is the attention mechanism, which compares the reference features to each candidate patch in the auxiliary sources. It assigns an attention score to each patch based on its similarity to the reference. The final representation is a weighted sum of these patches, effectively "finding" the correctly aligned object features automatically.

Experimental Results and Discussion

The model was evaluated using a dataset of 48,063 street trees across 40 species. The data included 1-foot resolution RGB, 8-band WorldView-2 multispectral data, and LiDAR-derived Digital Surface Models.

Comparison with Baselines

MRAN was compared against the standard "Feature Concatenation" approach, where features from different sources are simply stacked together assuming perfect alignment.

The standard feature concatenation baseline where sources are merged after feature extraction, assuming perfect alignment.
  • 18-Class Setting: MRAN achieved 64.2% accuracy, a 13% relative improvement over the baseline.
  • 40-Class Setting: MRAN reached 47.3% accuracy, a 14.3% relative improvement.

Key Insights

  • Spectral Importance: Despite lower spatial resolution, multispectral data proved more critical for species identification than high-resolution RGB texture.
  • Data Efficiency: MRAN demonstrated superior learning efficiency, achieving higher accuracy with only 20% of the training data than the baseline could achieve with 100%. This suggests the attention mechanism effectively filters out background noise caused by misalignment.

Conclusion

The Multisource Region Attention Network provides a robust solution for fine-grained recognition in urban environments. By shifting the paradigm from rigid image registration to dynamic attention-based selection, the model overcomes the geometric distortions common in multisensor remote sensing. Future work will explore the application of this framework to other small-scale objects, such as vehicles and specific building architectures, across diverse urban landscapes.

Summary updated

Abstract

Fine-grained object recognition concerns the identification of the type of an object among a large number of closely related sub-categories. Multisource data analysis, that aims to leverage the complementary spectral, spatial, and structural information embedded in different sources, is a promising direction towards solving the fine-grained recognition problem that involves low between-class variance, small training set sizes for rare classes, and class imbalance. However, the common assumption of co-registered sources may not hold at the pixel level for small objects of interest. We present a novel methodology that aims to simultaneously learn the alignment of multisource data and the classification model in a unified framework. The proposed method involves a multisource region attention network that computes per-source feature representations, assigns attention scores to candidate regions sampled around the expected object locations by using these representations, and classifies the objects by using an attention-driven multisource representation that combines the feature representations and the attention scores from all sources. All components of the model are realized using deep neural networks and are learned in an end-to-end fashion. Experiments using RGB, multispectral, and LiDAR elevation data for classification of street trees showed that our approach achieved 64.2% and 47.3% accuracies for the 18-class and 40-class settings, respectively, which correspond to 13% and 14.3% improvement relative to the commonly used feature concatenation approach from multiple sources.
BibTeX
@article{sumbul2019multisource,
  author = {Gencer Sumbul and Ramazan Gokberk Cinbis and Selim Aksoy},
  title = {{Multisource Region Attention Network for Fine-Grained Object Recognition in Remote Sensing Imagery}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2019},
  url = {https://openreview.net/forum?id=g6qFRRC1TI},
}
Figure