Abstract
Fine-grained object recognition in remote sensing aims to identify specific sub-categories, such as tree species, within broader classes. This task is challenging due to low between-class variance and the inherent misalignment of multisource data (e.g., RGB, multispectral, and LiDAR) at the pixel level. We present the Multisource Region Attention Network (MRAN), a unified framework that simultaneously learns data alignment and classification. By employing a region attention mechanism, MRAN assigns scores to candidate patches across different sensors to create a fused, aligned representation. Experiments on a street tree dataset demonstrate that MRAN achieves significant accuracy improvements—up to 14.3%—over traditional feature concatenation methods.
The Challenge of Fine-Grained Recognition
Traditional remote sensing classification often distinguishes between broad land-cover types like "water" or "forest." Fine-grained recognition pushes these boundaries by attempting to identify dozens of closely related species or object types. This task faces two primary hurdles:
- Visual Similarity: Sub-categories often share nearly identical spatial and spectral characteristics.
- Data Misalignment: While combining RGB, multispectral (MS), and LiDAR data provides a richer feature set, these sensors are rarely perfectly co-registered. Even a shift of a few pixels can cause a model to fuse features from a tree in one source with a sidewalk in another, leading to classification errors.
The MRAN Framework
The proposed Multisource Region Attention Network (MRAN) addresses misalignment by treating it as a learning problem rather than a preprocessing step. The architecture consists of a reference branch and multiple auxiliary branches.
Region Attention Mechanism
In the MRAN model, one source (typically high-resolution RGB) serves as the reference. The model extracts features from a verified location in this reference. For auxiliary sources like MS and LiDAR, the model samples a grid of region proposals around the expected location.
The core innovation is the attention mechanism, which compares the reference features to each candidate patch in the auxiliary sources. It assigns an attention score to each patch based on its similarity to the reference. The final representation is a weighted sum of these patches, effectively "finding" the correctly aligned object features automatically.
Experimental Results and Discussion
The model was evaluated using a dataset of 48,063 street trees across 40 species. The data included 1-foot resolution RGB, 8-band WorldView-2 multispectral data, and LiDAR-derived Digital Surface Models.
Comparison with Baselines
MRAN was compared against the standard "Feature Concatenation" approach, where features from different sources are simply stacked together assuming perfect alignment.
- 18-Class Setting: MRAN achieved 64.2% accuracy, a 13% relative improvement over the baseline.
- 40-Class Setting: MRAN reached 47.3% accuracy, a 14.3% relative improvement.
Key Insights
- Spectral Importance: Despite lower spatial resolution, multispectral data proved more critical for species identification than high-resolution RGB texture.
- Data Efficiency: MRAN demonstrated superior learning efficiency, achieving higher accuracy with only 20% of the training data than the baseline could achieve with 100%. This suggests the attention mechanism effectively filters out background noise caused by misalignment.
Conclusion
The Multisource Region Attention Network provides a robust solution for fine-grained recognition in urban environments. By shifting the paradigm from rigid image registration to dynamic attention-based selection, the model overcomes the geometric distortions common in multisensor remote sensing. Future work will explore the application of this framework to other small-scale objects, such as vehicles and specific building architectures, across diverse urban landscapes.
Summary updated