Multi-modal land cover mapping of remote sensing images using pyramid attention and gated fusion networks

Qinghui Liu, Michael Kampffmeyer, Robert Jenssen, Arnt-Børre SalbergInternational Journal of Remote Sensing 2021 · DOI · OpenReview

Abstract

Multi-modality data is becoming readily available in remote sensing (RS) and can provide complementary information about the Earth’s surface. Effective fusion of multi-modal information is thus important for various applications in RS, but also very challenging due to large domain differences, noise, and redundancies. There is a lack of effective and scalable fusion techniques for bridging multiple modality encoders and fully exploiting complementary information. To this end, we propose a new multi-modality network (MultiModNet) for land cover mapping of multi-modal remote sensing data based on a novel pyramid attention fusion (PAF) module and a gated fusion unit (GFU). The PAF module is designed to efficiently obtain rich fine-grained contextual representations from each modality with a built-in cross-level and cross-view attention fusion mechanism, and the GFU module utilizes a novel gating mechanism for early merging of features, thereby diminishing hidden redundancies and noise. This enables supplementary modalities to effectively extract the most valuable and complementary information for late feature fusion. Extensive experiments on two representative RS benchmark datasets demonstrate the effectiveness, robustness, and superiority of the MultiModNet for multi-modal land cover classification.

Introduction

In remote sensing, researchers often have access to multiple "modalities" for the same geographic location, such as standard RGB or Infrared (IRRG) images alongside Digital Surface Models (DSM) providing height information. While theoretically beneficial, fusing these sources is difficult because different modalities possess distinct "languages" (domains), varying noise levels, and significant redundancies.

Traditional fusion methods often rely on simple concatenation or addition, which can lead to models being confused by noisy data or failing to capture fine-grained details where one modality should have complemented the other. MultiModNet addresses these challenges with a scalable architecture that intelligently filters and fuses information.

Proposed Method: MultiModNet

The MultiModNet framework introduces two primary architectural innovations designed to bridge multiple modality encoders efficiently.

The MultiModNet architecture utilizes Gated Fusion Units (GFU) and Pyramid Attention Fusion (PAF) modules to scale across multiple input modalities.

Pyramid Attention Fusion (PAF) Module

Unlike standard self-attention mechanisms that are computationally expensive, the PAF module employs a "cross-level" approach. It extracts features from three different scales of the encoder to build a "tall" attention matrix. This allows the model to capture high-resolution details using context from lower-resolution, high-abstraction layers. Furthermore, it includes a cross-view mechanism to ensure robustness against the arbitrary rotations common in aerial imagery.

Gated Fusion Unit (GFU)

The GFU acts as an intelligent filter. It uses the primary modality (e.g., an IRRG image) to gate the secondary modality (e.g., a DSM). If the secondary data is noisy or redundant in a specific area, the GFU suppresses it; if it provides valuable complementary information, such as height data missing from the 2D image, the GFU allows it to pass through. This modular design allows the network to scale to any number of modalities by adding more GFU/PAF blocks.

Experimental Results

Datasets and Performance

The model was evaluated on two major benchmarks:

  1. ISPRS Vaihingen Dataset: An urban classification task where MultiModNet achieved a state-of-the-art mF1 score of 90.7%, notably improving the identification of small objects like cars by 2.5%.
  2. Agriculture-Vision Dataset: A large-scale farmland dataset where the model reached 48.2% mIoU, a 4.2% improvement over previous benchmarks.
Qualitative comparison of land cover mapping results, showing MultiModNet's superior ability to identify fine-grained details like cars and building edges compared to baseline methods.

Efficiency and Robustness

MultiModNet is not only more accurate but also more efficient. The "PAGNet-b" variant is twice as fast in training and inference compared to competitors while maintaining a smaller parameter footprint (6.14M).

Crucially, the model demonstrates extreme noise tolerance. When secondary modality data (DSM) was intentionally corrupted or removed during testing, MultiModNet's performance dropped by less than 1%. This suggests the GFU successfully learns to rely on the primary modality when the secondary sensor fails, a vital trait for real-world deployment.

Conclusion

MultiModNet provides a robust and scalable solution for multi-modal remote sensing. By utilizing primary-guided filtering and cross-level attention, it overcomes the traditional pitfalls of multi-sensor data fusion. While the current model relies heavily on the integrity of the primary modality, future work will explore bidirectional gating to allow modalities to support each other more symmetrically and extend the framework to other domains like autonomous driving and medical imaging.

Summary updated

Abstract

Multi-modality data is becoming readily available in remote sensing (RS) and can provide complementary information about the Earth's surface. Effective fusion of multi-modal information is thus important for various applications in RS, but also very challenging due to large domain differences, noise, and redundancies. There is a lack of effective and scalable fusion techniques for bridging multiple modality encoders and fully exploiting complementary information. To this end, we propose a new multi-modality network (MultiModNet) for land cover mapping of multi-modal remote sensing data based on a novel pyramid attention fusion (PAF) module and a gated fusion unit (GFU). The PAF module is designed to efficiently obtain rich fine-grained contextual representations from each modality with a built-in cross-level and cross-view attention fusion mechanism, and the GFU module utilizes a novel gating mechanism for early merging of features, thereby diminishing hidden redundancies and noise. This enables supplementary modalities to effectively extract the most valuable and complementary information for late feature fusion. Extensive experiments on two representative RS benchmark datasets demonstrate the effectiveness, robustness, and superiority of the MultiModNet for multi-modal land cover classification.
BibTeX
@article{liu2021multi,
  author = {Qinghui Liu and Michael Kampffmeyer and Robert Jenssen and Arnt-Børre Salberg},
  title = {{Multi-modal land cover mapping of remote sensing images using pyramid attention and gated fusion networks}},
  journal = {International Journal of Remote Sensing},
  year = {2021},
  doi = {10.1080/01431161.2022.2098078},
}
Figure