AFM-Net: Advanced Fusing Hierarchical CNN Visual Priors with Global Sequence Modeling for Remote Sensing Image Scene Classification

Yuanhao Tang, Xuechao Zou, Zhengpei Hu, Junliang Xing, Chengkun Zhang, Jianqiang HuangarXiv 2025 · DOI · arXiv

Abstract

Remote sensing image scene classification (RSIC) faces significant challenges due to complex spatial structures and high intra-class diversity. This paper introduces AFM-Net, a novel hybrid architecture that combines the local texture-capturing strengths of Convolutional Neural Networks (CNNs) with the efficient global context modeling of Mamba. By employing a hierarchical fusion module and a task-aware Mixture-of-Experts (MoE) classifier head, AFM-Net achieves state-of-the-art accuracy across major benchmarks while maintaining significantly higher computational efficiency than standalone Transformer or Mamba-based backbones.

Introduction

Remote sensing images contain objects at multiple scales and intricate spatial arrangements, making classification difficult. Traditional CNNs excel at extracting fine-grained local textures but often fail to capture long-range spatial interactions. Conversely, modern Transformers and Mamba-based models provide excellent global context but can be computationally expensive or overlook critical local details. AFM-Net addresses this bottleneck by "marrying" these two paradigms into a unified, efficient framework.

Methodology

AFM-Net is built upon three core components designed to balance detail and context:

1. Parallel Heterogeneous Encoder

Rather than relying on a single backbone, the model utilizes two parallel encoders. A CNN-based local encoder (ResNet-based) extracts multi-scale textures, while a shallow Mamba-based encoder serves as a lightweight global context enhancer. This configuration ensures global modeling remains efficient without the overhead of a deep, standalone Mamba backbone.

2. Hierarchical Fusion Module

This module bridges the gap between the local and global encoders using a top-down aggregation strategy. It utilizes DAMF-Blocks (Densely connected dual-attention multi-scale fusion) to progressively combine features, ensuring the final representation is both detailed and context-aware.

The hierarchical fusion architecture showing the integration of local (L) and global (G) features through DAMF blocks.

3. Mixture-of-Experts (MoE) Classifier Head

To handle high intra-class variance, the model employs an MoE head consisting of a "shared expert" for common patterns and multiple "routed experts." A dynamic router dispatches features to the most suitable experts, allowing for specialized decision-making with minimal additional computation.

Experiments and Results

AFM-Net was evaluated on three standard benchmarks: UC Merced, AID, and NWPU-RESISC45.

Performance and Efficiency

The model set new performance benchmarks, achieving 96.92% on UC Merced, 93.76% on AID, and 95.54% on NWPU-RESISC45. Crucially, AFM-Net demonstrates superior efficiency, requiring only 11.67G FLOPs—over 12x fewer than the 146.92G required by RSMamba-L—while delivering higher accuracy.

Accuracy vs. GFLOPs comparison showing AFM-Net (Ours) achieving superior accuracy with significantly lower computational cost compared to other SOTA models.

Ablation Studies

Ablation tests confirmed the necessity of the hybrid approach; removing the local CNN encoder resulted in a catastrophic 5.58% drop in F1-score, proving that global modeling alone is insufficient for high-resolution remote sensing tasks.

Discussion

Key insights from the development of AFM-Net include:

  • Mamba as an Enhancer: Mamba is most effective as a lightweight auxiliary branch rather than a deep backbone, preserving the inductive bias of CNNs while adding long-range dependency modeling.
  • Dynamic Routing for Diversity: The MoE head effectively manages the "messy" nature of real-world aerial data by specializing knowledge across different experts.

Conclusion

AFM-Net provides a highly efficient and accurate solution for RSIC by successfully integrating hierarchical CNN priors with global sequence modeling. Future research will focus on adapting this hierarchical fusion and MoE framework for more complex tasks such as object detection and semantic segmentation in remote sensing imagery.

Summary updated

Abstract

Remote sensing image scene classification remains a challenging task, primarily due to the complex spatial structures and multi-scale characteristics of ground objects. Although CNN-based methods excel at extracting local inductive biases, and Mamba-based approaches demonstrate impressive capabilities in efficiently capturing global sequential context, relying on a single paradigm restricts the model's ability to simultaneously characterize fine-grained textures and complex spatial structures. To tackle this, we propose a parallel heterogeneous encoder, a hierarchical fusion module designed to achieve effective local-global co-representation. It consists of two parallel pathways: a local visual encoder for extracting multi-scale local visual features, and a global visual encoder for capturing efficient global visual features. The core innovation lies in its hierarchical fusion module, which progressively aggregates multi-scale features from both pathways, enabling dynamic cross-level feature interaction and contextual reconstruction to produce highly discriminative representations. These fused features are then adaptively routed through a mixture-of-experts classifier head, which dynamically dispatches them to the most suitable experts for fine-grained scene recognition. Experiments on AID, NWPU-RESISC45, and UC Merced show that our model achieves 93.72%, 95.54%, and 96.92% accuracy, surpassing SOTA methods with an optimal balance of performance and efficiency. Code is available at https://anonymous.4open.science/r/classification-41DF.
BibTeX
@misc{tang2025afm,
  author = {Yuanhao Tang and Xuechao Zou and Zhengpei Hu and Junliang Xing and Chengkun Zhang and Jianqiang Huang},
  title = {{AFM-Net: Advanced Fusing Hierarchical CNN Visual Priors with Global Sequence Modeling for Remote Sensing Image Scene Classification}},
  year = {2025},
  eprint = {2510.27155},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.27155},
}
Figure