Abstract
Multi-modality data is becoming readily available in remote sensing (RS) and can provide complementary information about the Earth’s surface. Effective fusion of multi-modal information is thus important for various applications in RS, but also very challenging due to large domain differences, noise, and redundancies. There is a lack of effective and scalable fusion techniques for bridging multiple modality encoders and fully exploiting complementary information. To this end, we propose a new multi-modality network (MultiModNet) for land cover mapping of multi-modal remote sensing data based on a novel pyramid attention fusion (PAF) module and a gated fusion unit (GFU). The PAF module is designed to efficiently obtain rich fine-grained contextual representations from each modality with a built-in cross-level and cross-view attention fusion mechanism, and the GFU module utilizes a novel gating mechanism for early merging of features, thereby diminishing hidden redundancies and noise. This enables supplementary modalities to effectively extract the most valuable and complementary information for late feature fusion. Extensive experiments on two representative RS benchmark datasets demonstrate the effectiveness, robustness, and superiority of the MultiModNet for multi-modal land cover classification.
Introduction
In remote sensing, researchers often have access to multiple "modalities" for the same geographic location, such as standard RGB or Infrared (IRRG) images alongside Digital Surface Models (DSM) providing height information. While theoretically beneficial, fusing these sources is difficult because different modalities possess distinct "languages" (domains), varying noise levels, and significant redundancies.
Traditional fusion methods often rely on simple concatenation or addition, which can lead to models being confused by noisy data or failing to capture fine-grained details where one modality should have complemented the other. MultiModNet addresses these challenges with a scalable architecture that intelligently filters and fuses information.
Proposed Method: MultiModNet
The MultiModNet framework introduces two primary architectural innovations designed to bridge multiple modality encoders efficiently.
Pyramid Attention Fusion (PAF) Module
Unlike standard self-attention mechanisms that are computationally expensive, the PAF module employs a "cross-level" approach. It extracts features from three different scales of the encoder to build a "tall" attention matrix. This allows the model to capture high-resolution details using context from lower-resolution, high-abstraction layers. Furthermore, it includes a cross-view mechanism to ensure robustness against the arbitrary rotations common in aerial imagery.
Gated Fusion Unit (GFU)
The GFU acts as an intelligent filter. It uses the primary modality (e.g., an IRRG image) to gate the secondary modality (e.g., a DSM). If the secondary data is noisy or redundant in a specific area, the GFU suppresses it; if it provides valuable complementary information, such as height data missing from the 2D image, the GFU allows it to pass through. This modular design allows the network to scale to any number of modalities by adding more GFU/PAF blocks.
Experimental Results
Datasets and Performance
The model was evaluated on two major benchmarks:
- ISPRS Vaihingen Dataset: An urban classification task where MultiModNet achieved a state-of-the-art mF1 score of 90.7%, notably improving the identification of small objects like cars by 2.5%.
- Agriculture-Vision Dataset: A large-scale farmland dataset where the model reached 48.2% mIoU, a 4.2% improvement over previous benchmarks.
Efficiency and Robustness
MultiModNet is not only more accurate but also more efficient. The "PAGNet-b" variant is twice as fast in training and inference compared to competitors while maintaining a smaller parameter footprint (6.14M).
Crucially, the model demonstrates extreme noise tolerance. When secondary modality data (DSM) was intentionally corrupted or removed during testing, MultiModNet's performance dropped by less than 1%. This suggests the GFU successfully learns to rely on the primary modality when the secondary sensor fails, a vital trait for real-world deployment.
Conclusion
MultiModNet provides a robust and scalable solution for multi-modal remote sensing. By utilizing primary-guided filtering and cross-level attention, it overcomes the traditional pitfalls of multi-sensor data fusion. While the current model relies heavily on the integrity of the primary modality, future work will explore bidirectional gating to allow modalities to support each other more symmetrically and extend the framework to other domains like autonomous driving and medical imaging.
Summary updated