TL;DR
This paper introduces HyFormer, a novel hybrid deep learning framework that combines Convolutional Neural Networks (CNNs) and Transformers to improve pixel-level land cover classification in multispectral remote sensing images. By effectively fusing spectral and spatial information, HyFormer achieves state-of-the-art accuracy, outperforming existing methods like ViT and CNNs alone.
What Problem Does This Paper Address?
Remote sensing (RS) image classification is crucial for various applications, but accurately classifying land cover at the pixel level presents challenges. Traditional Convolutional Neural Networks (CNNs) often struggle to effectively process the sequential nature of spectral information in RS data and may not adequately capture global contextual relationships. While Transformers excel at sequence modeling and global feature extraction, they can be computationally intensive and might not fully leverage local spectral details. This paper aims to bridge this gap by developing a hybrid architecture that harnesses the strengths of both CNNs and Transformers for more robust and accurate pixel-level land cover classification.
Methods
HyFormer employs a multi-stage approach:
- 1D Pixel Sequence to 3D Feature Matrix: Initially, 1D pixel spectral sequences are processed by a fully connected (FC) layer to enhance dimensionality. This is then reshaped into a 3D spectral feature matrix, enabling the application of 2D CNNs for pixel-level classification, which is a novel way to adapt 2D CNNs to pixel-wise spectral data.
- CNN Feature Extraction: A shallow 2D CNN architecture, incorporating residual units and Squeeze-and-Excitation (SE) modules, is used to extract features at three different levels.
- Hybrid Fusion with Transformer: The extracted CNN features are combined with the original spectral information and positional embeddings to form the input for a Transformer encoder. This allows the Transformer to leverage both local spectral-spatial features from the CNN and global context.
- Skip Connections: Short hopping links are utilized between adjacent Transformer encoders to facilitate feature fusion and mitigate information loss, enhancing the flow of learned representations.
- Pixel Classification: Finally, an MLP head takes the fused features to produce the pixel-level classification results.
Datasets
The study utilizes Sentinel-2 multispectral images covering two regions in Zhejiang Province, China:
- Changxing County: Data from July 27, 2022, with a 10m resolution. A sample library of 15,391 samples across 8 categories was created.
- Nanxun District: Data from July 27, 2022, with a 10m resolution. A sample library of 18,263 samples across 8 categories was used.
Additionally, historical Sentinel-2 data from 2018 and 2020 for Changxing County were used for land use change analysis.
Key Insights and Ideas
- Bridging CNN and Transformer Strengths: The core idea is to use CNNs to extract rich local spectral-spatial features and then feed these into a Transformer for global context modeling, creating a synergistic effect.
- Adapting 2D CNNs for Pixel Sequences: Reshaping 1D pixel sequences into 3D matrices is a clever way to apply powerful 2D CNN architectures to pixel-level classification tasks, expanding their applicability.
- Effective Feature Fusion: The combination of CNN features, positional embeddings, and Transformer's self-attention mechanism allows for a more comprehensive understanding of the spectral information.
- Importance of Skip Connections: Short skip connections are crucial for effectively fusing features across different Transformer encoder layers, preventing information degradation.
- Dynamic Land Use Analysis: The framework is also demonstrated to be effective for analyzing land use changes over time, highlighting its practical utility.
Key Results
- Superior Accuracy: HyFormer achieved the highest overall accuracy (OA) on both study areas: 95.37% in Changxing County and 95.4% in Nanxun District.
- Outperforming Baselines: HyFormer significantly outperformed traditional methods like SVM (89.31% OA in Changxing) and deep learning models like ViT (94.15% OA in Changxing, 94.69% OA in Nanxun), 1D CNN, and RNN.
- Ablation Study: The ablation study confirmed the effectiveness of each component. The combination of CNN and Transformer (HyFormer) showed a 1.12% improvement over ViT alone. Adding skip connections further boosted performance.
- Land Use Change Analysis: The model demonstrated robust performance across different years (2018, 2020, 2022), enabling accurate land use change analysis, particularly for pond and rice cultivation areas.
Limitations and Future Work
- Generalizability: The authors acknowledge that HyFormer was primarily tested on multispectral images and has not yet been experimented with panchromatic and hyperspectral images. Future work could explore its performance on these data types.
- Complexity Reduction: While effective, the Transformer component can be computationally intensive. Future research could focus on reducing the complexity of the Transformer while maintaining high performance.
- Feature Diversity: Enhancing feature diversity by exploring different methods for converting 1D pixel sequences to 2D or 3D feature maps is suggested for future improvements.
- Index Integration: Incorporating vegetation indices, building indices, and other remote sensing indices alongside the original bands could potentially further improve classification differentiation.
Summary updated