Weakly Supervised Instance Attention for Multisource Fine-Grained Object Recognition with an Application to Tree Species Classification

Bulut Aygünes, Ramazan Gökberk Cinbis, Selim AksoyarXiv 2021 · DOI · OpenReview

TL;DR

This paper tackles the challenge of fine-grained object recognition in multisource remote sensing data, where even small registration errors between different data types (like RGB, multispectral, and LiDAR) can lead to significant uncertainty. They propose a weakly supervised "instance attention" model that learns to jointly localize and classify objects within larger image patches, and extend this to fuse information from multiple sources, achieving state-of-the-art results in tree species classification.

What Problem Does This Paper Address?

Fine-grained object recognition, which aims to distinguish between very similar subcategories (e.g., different tree species), is inherently difficult. This challenge is amplified when using multiple data sources (like RGB, multispectral, and LiDAR) for remote sensing. While these sources offer complementary information, they are often not perfectly aligned (registered). Even minor misalignments can introduce substantial uncertainty, especially for small objects like trees, making it hard to know which pixels in one data source correspond to the object of interest in another. Traditional supervised learning methods struggle with this "location uncertainty" and the potential for class imbalance in fine-grained datasets. This paper addresses this by developing a weakly supervised approach that doesn't require precise object localization within the input images.

Methods

The core of the proposed method is a Weakly Supervised Instance Attention (WSIA) model. Instead of relying on exact bounding boxes, WSIA takes larger image patches where the object is known to be present but its exact location is unknown. It then uses a deep learning architecture with two parallel branches: one for localization and one for classification. The localization branch learns to score candidate regions within the patch, while the classification branch predicts the object's class. By jointly optimizing these, the model learns to attend to relevant regions.

The paper then extends this single-source WSIA model to a multisource setting. They introduce four novel fusion schemes, leveraging a "reference source" (assumed to have minimal location uncertainty, typically high-resolution RGB) to guide the fusion of "additional sources" (which may have higher uncertainty, like multispectral or LiDAR). These fusion methods operate at different levels:

  • Probability-Level-Fusion: Averages the predicted probabilities from each source.
  • Logit-Level-Fusion: Combines the raw output scores (logits) before the final softmax, allowing for more flexible weighting.
  • Feature-Level-Fusion: Concatenates features from the reference source with features from the additional sources at an earlier stage, allowing for richer interaction. This proved to be the most effective.
  • Pixel-Level-Fusion: Concatenates reference features directly with the input pixels of the additional sources.

Datasets

The experiments were conducted on a publicly available dataset of street trees, specifically the same dataset used by Sumbul et al. (2019). This dataset contains:

  • Aerial RGB images: High spatial resolution (1 foot).
  • 8-band WorldView-2 Multispectral (MS) images: Lower spatial resolution (2 meters).
  • LiDAR-based Digital Surface Model (DSM): Lower spatial resolution (3 feet).

The dataset comprises 48,063 instances of street trees across 40 different classes, exhibiting significant class imbalance. The ground truth labels were obtained from point GIS data, with the RGB images serving as the reference source due to their higher resolution and more reliable georeferencing.

Key Insights and Ideas

  • Weakly Supervised Learning for Location Uncertainty: The core idea of using larger image patches and a weakly supervised approach is crucial for handling registration errors in multisource data, especially for small objects.
  • Instance Attention for Joint Localization and Classification: The WSIA model's ability to simultaneously learn where an object is and what it is, even without explicit bounding boxes, is a key enabler.
  • Multi-Level Fusion Strategies: The exploration of different fusion levels (probability, logit, feature, pixel) reveals that integrating information at higher levels (feature-level) is more beneficial for this problem.
  • Importance of Model Capacity: Comparing models across various parameter complexities is vital for fair evaluation, as a model's performance can significantly improve with increased capacity, as demonstrated by the Feature-Level-Fusion results.
  • Complementarity of Data Sources: While RGB is a strong reference, multispectral data provides essential spectral information that significantly boosts performance, even outperforming RGB alone for many classes. LiDAR's contribution, while present, is less impactful than MS for this specific fine-grained task.

Key Results

  • State-of-the-Art Performance: The proposed multisource instance attention models significantly outperform existing state-of-the-art methods.
  • Feature-Level-Fusion Dominance: The Feature-Level-Fusion method achieved the best results, reaching 53.0% accuracy for the 40 tree species classification task using RGB, MS, and LiDAR data. This represents a 5.7% improvement over the best performing baseline.
  • Impact of Model Capacity: Increasing the model capacity for Feature-Level-Fusion led to a further 6.3% improvement, reaching 58.0% accuracy for the RGB & MS setting.
  • Single-Source Improvements: The single-source instance attention model also showed substantial gains over baseline methods for MS (40.6% to 48.3%) and LiDAR (21.2% to 25.3%).
  • Localization Benefits: Ablation studies confirmed that the localization branch within the instance attention model significantly improves performance.

Limitations and Future Work

  • Generalizability: The authors suggest future work should explore the generalizability of their proposed method on other multisource fine-grained datasets.
  • Other Uncertainties: Extending the model to handle other types of uncertainties, such as temporal changes, is a promising direction.
  • Performance Metrics: The use of additional metrics like the Kappa coefficient for performance evaluation is suggested.
  • LiDAR Contribution: While LiDAR contributes, its impact is less pronounced than MS. Future work could investigate ways to better leverage LiDAR data for fine-grained tasks.
  • Pixel-Level Fusion Drop: The performance drop in Pixel-Level-Fusion suggests that combining high-level reference features with low-level pixel data might not be as effective as integrating them at later stages.

Summary updated

Abstract

Multisource image analysis that leverages complementary spectral, spatial, and structural information benefits fine-grained object recognition that aims to classify an object into one of many similar subcategories. However, for multisource tasks that involve relatively small objects, even the smallest registration errors can introduce high uncertainty in the classification process. We approach this problem from a weakly supervised learning perspective in which the input images correspond to larger neighborhoods around the expected object locations where an object with a given class label is present in the neighborhood without any knowledge of its exact location. The proposed method uses a single-source deep instance attention model with parallel branches for joint localization and classification of objects, and extends this model into a multisource setting where a reference source that is assumed to have no location uncertainty is used to aid the fusion of multiple sources in four different levels: probability level, logit level, feature level, and pixel level. We show that all levels of fusion provide higher accuracies compared to the state-of-the-art, with the best performing method of feature-level fusion resulting in 53% accuracy for the recognition of 40 different types of trees, corresponding to an improvement of 5.7% over the best performing baseline when RGB, multispectral, and LiDAR data are used. We also provide an in-depth comparison by evaluating each model at various parameter complexity settings, where the increased model capacity results in a further improvement of 6.3% over the default capacity setting.
BibTeX
@misc{aygunes2021weakly,
  author = {Bulut Aygünes and Ramazan Gökberk Cinbis and Selim Aksoy},
  title = {{Weakly Supervised Instance Attention for Multisource Fine-Grained Object Recognition with an Application to Tree Species Classification}},
  year = {2021},
  eprint = {2105.10983},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.10983},
}
Figure