Abstract
As ground objects become increasingly complex, classification results from single-source remote sensing data often fall short of application requirements. We propose a simple yet effective attention fusion model based on a Disentangled Non-local (DNL) network for joint hyperspectral imaging (HSI) and LiDAR data classification. Our model utilizes a multiscale module and a convolutional neural network (CNN) to capture spectral and spatial characteristics, respectively. These features are fused and processed through a DNL module, which separates traditional self-attention into pairwise and unary terms. Experiments on the Houston dataset demonstrate that the proposed network outperforms several state-of-the-art baselines.
Introduction
In remote sensing, relying on a single data source often leads to classification errors. Hyperspectral images (HSI) provide rich spectral detail, allowing for material differentiation, but often lack the spatial resolution or elevation data needed to distinguish between objects with similar spectral signatures, such as a flat green roof versus a patch of grass. Conversely, LiDAR provides precise 3D elevation data but lacks the spectral depth to identify specific vegetation types.
While fusing these data types is logical, traditional deep learning methods like standard CNNs face challenges:
- Limited Receptive Fields: Standard convolutions focus on local neighborhoods, missing long-range spatial patterns.
- Coupled Attention: Standard Non-Local (self-attention) blocks mix global context and pixel-to-pixel relationships, often resulting in "blurry" object boundaries.
This paper introduces a Disentangled Non-Local (DNL) network designed to address these limitations by splitting self-attention into independent "pairwise" and "unary" terms.
Methodology
The proposed architecture follows a multi-stage process:
Feature Extraction
- HSI Branch: Employs a multiscale residual module using various filter sizes ($1\times1, 3\times3, 5\times5, 7\times7$) to capture spectral features across different spatial scales.
- LiDAR Branch: Uses a standard CNN to extract elevation and structural features.
Feature Fusion and the DNL Module
The HSI and LiDAR features are combined through addition and pooling to create a fused feature set ($F$). The core innovation is the DNL module, which disentangles the traditional dot-product attention.
The DNL module splits attention into:
- Unary Term: Learns the global influence of a pixel (e.g., general patterns of roads across the image).
- Pairwise Term: Learns specific relationships between neighboring pixels.
The authors found that using LiDAR features as the Query, HSI features as the Key, and the Fused features as the Value yielded optimal results.
Experimental Results
The model was evaluated on the Houston dataset, which contains 144 spectral bands and 2.5-meter spatial resolution across 15 land-cover categories.
Quantitative Performance
The DNL-based approach achieved superior performance compared to baselines:
- Overall Accuracy (OA): 93.74%
- Average Accuracy (AA): 94.78%
- Kappa Coefficient: 93.41%
Qualitative Analysis
Visual inspection of the classification maps reveals that the DNL method produces cleaner, more continuous regions with significantly fewer "salt-and-pepper" noise errors than traditional CNN or SVM approaches.
Key Insights
- Disentanglement: Separating unary and pairwise terms prevents interference, leading to sharper classification boundaries.
- Cross-Modal Roles: Assigning LiDAR to "query" HSI data allows the model to correlate spectral signatures with physical heights effectively.
- Multiscale Integration: Using multiple kernel sizes ensures the detection of both small objects (trees) and large structures (parking lots).
Conclusion
The DNL network offers a robust solution for multi-source remote sensing data fusion. While highly effective, future work could address the computational complexity of non-local blocks and test the model's generalization across more diverse geographic environments.
Summary updated