Abstract
Remote sensing image scene classification (RSIC) faces significant challenges due to complex spatial structures and high intra-class diversity. This paper introduces AFM-Net, a novel hybrid architecture that combines the local texture-capturing strengths of Convolutional Neural Networks (CNNs) with the efficient global context modeling of Mamba. By employing a hierarchical fusion module and a task-aware Mixture-of-Experts (MoE) classifier head, AFM-Net achieves state-of-the-art accuracy across major benchmarks while maintaining significantly higher computational efficiency than standalone Transformer or Mamba-based backbones.
Introduction
Remote sensing images contain objects at multiple scales and intricate spatial arrangements, making classification difficult. Traditional CNNs excel at extracting fine-grained local textures but often fail to capture long-range spatial interactions. Conversely, modern Transformers and Mamba-based models provide excellent global context but can be computationally expensive or overlook critical local details. AFM-Net addresses this bottleneck by "marrying" these two paradigms into a unified, efficient framework.
Methodology
AFM-Net is built upon three core components designed to balance detail and context:
1. Parallel Heterogeneous Encoder
Rather than relying on a single backbone, the model utilizes two parallel encoders. A CNN-based local encoder (ResNet-based) extracts multi-scale textures, while a shallow Mamba-based encoder serves as a lightweight global context enhancer. This configuration ensures global modeling remains efficient without the overhead of a deep, standalone Mamba backbone.
2. Hierarchical Fusion Module
This module bridges the gap between the local and global encoders using a top-down aggregation strategy. It utilizes DAMF-Blocks (Densely connected dual-attention multi-scale fusion) to progressively combine features, ensuring the final representation is both detailed and context-aware.
3. Mixture-of-Experts (MoE) Classifier Head
To handle high intra-class variance, the model employs an MoE head consisting of a "shared expert" for common patterns and multiple "routed experts." A dynamic router dispatches features to the most suitable experts, allowing for specialized decision-making with minimal additional computation.
Experiments and Results
AFM-Net was evaluated on three standard benchmarks: UC Merced, AID, and NWPU-RESISC45.
Performance and Efficiency
The model set new performance benchmarks, achieving 96.92% on UC Merced, 93.76% on AID, and 95.54% on NWPU-RESISC45. Crucially, AFM-Net demonstrates superior efficiency, requiring only 11.67G FLOPs—over 12x fewer than the 146.92G required by RSMamba-L—while delivering higher accuracy.
Ablation Studies
Ablation tests confirmed the necessity of the hybrid approach; removing the local CNN encoder resulted in a catastrophic 5.58% drop in F1-score, proving that global modeling alone is insufficient for high-resolution remote sensing tasks.
Discussion
Key insights from the development of AFM-Net include:
- Mamba as an Enhancer: Mamba is most effective as a lightweight auxiliary branch rather than a deep backbone, preserving the inductive bias of CNNs while adding long-range dependency modeling.
- Dynamic Routing for Diversity: The MoE head effectively manages the "messy" nature of real-world aerial data by specializing knowledge across different experts.
Conclusion
AFM-Net provides a highly efficient and accurate solution for RSIC by successfully integrating hierarchical CNN priors with global sequence modeling. Future research will focus on adapting this hierarchical fusion and MoE framework for more complex tasks such as object detection and semantic segmentation in remote sensing imagery.
Summary updated