Disentangled Non-Local Network for Hyperspectral and LiDAR Data Classification

Wenxia Liu, Feng Gao, Junyu Dong

Abstract

As ground objects become increasingly complex, classification results from single-source remote sensing data often fall short of application requirements. We propose a simple yet effective attention fusion model based on a Disentangled Non-local (DNL) network for joint hyperspectral imaging (HSI) and LiDAR data classification. Our model utilizes a multiscale module and a convolutional neural network (CNN) to capture spectral and spatial characteristics, respectively. These features are fused and processed through a DNL module, which separates traditional self-attention into pairwise and unary terms. Experiments on the Houston dataset demonstrate that the proposed network outperforms several state-of-the-art baselines.

Introduction

In remote sensing, relying on a single data source often leads to classification errors. Hyperspectral images (HSI) provide rich spectral detail, allowing for material differentiation, but often lack the spatial resolution or elevation data needed to distinguish between objects with similar spectral signatures, such as a flat green roof versus a patch of grass. Conversely, LiDAR provides precise 3D elevation data but lacks the spectral depth to identify specific vegetation types.

While fusing these data types is logical, traditional deep learning methods like standard CNNs face challenges:

  1. Limited Receptive Fields: Standard convolutions focus on local neighborhoods, missing long-range spatial patterns.
  2. Coupled Attention: Standard Non-Local (self-attention) blocks mix global context and pixel-to-pixel relationships, often resulting in "blurry" object boundaries.

This paper introduces a Disentangled Non-Local (DNL) network designed to address these limitations by splitting self-attention into independent "pairwise" and "unary" terms.

Methodology

The proposed architecture follows a multi-stage process:

Feature Extraction

  • HSI Branch: Employs a multiscale residual module using various filter sizes ($1\times1, 3\times3, 5\times5, 7\times7$) to capture spectral features across different spatial scales.
  • LiDAR Branch: Uses a standard CNN to extract elevation and structural features.

Feature Fusion and the DNL Module

The HSI and LiDAR features are combined through addition and pooling to create a fused feature set ($F$). The core innovation is the DNL module, which disentangles the traditional dot-product attention.

The architecture of the Disentangled Non-Local (DNL) module, showing the separation of unary and pairwise terms for cross-modal fusion.

The DNL module splits attention into:

  • Unary Term: Learns the global influence of a pixel (e.g., general patterns of roads across the image).
  • Pairwise Term: Learns specific relationships between neighboring pixels.

The authors found that using LiDAR features as the Query, HSI features as the Key, and the Fused features as the Value yielded optimal results.

Experimental Results

The model was evaluated on the Houston dataset, which contains 144 spectral bands and 2.5-meter spatial resolution across 15 land-cover categories.

Quantitative Performance

The DNL-based approach achieved superior performance compared to baselines:

  • Overall Accuracy (OA): 93.74%
  • Average Accuracy (AA): 94.78%
  • Kappa Coefficient: 93.41%

Qualitative Analysis

Visual inspection of the classification maps reveals that the DNL method produces cleaner, more continuous regions with significantly fewer "salt-and-pepper" noise errors than traditional CNN or SVM approaches.

Comparison of classification results on the Houston dataset. (a) HSI pseudocolor, (b) LiDAR DSM, (c) Ground-truth, and (d-i) results from various methods including the proposed DNL (i).

Key Insights

  • Disentanglement: Separating unary and pairwise terms prevents interference, leading to sharper classification boundaries.
  • Cross-Modal Roles: Assigning LiDAR to "query" HSI data allows the model to correlate spectral signatures with physical heights effectively.
  • Multiscale Integration: Using multiple kernel sizes ensures the detection of both small objects (trees) and large structures (parking lots).

Conclusion

The DNL network offers a robust solution for multi-source remote sensing data fusion. While highly effective, future work could address the computational complexity of non-local blocks and test the model's generalization across more diverse geographic environments.

Summary updated

Abstract

As the ground objects become increasingly complex, the classification results obtained by single source remote sensing data can hardly meet the application requirements. In order to tackle this limitation, we propose a simple yet effective attention fusion model based on Disentangled Non-local (DNL) network for hyperspectral and LiDAR data joint classification task. In this model, according to the spectral and spatial characteristics of HSI and LiDAR, a multiscale module and a convolutional neural network (CNN) are used to capture the spectral and spatial characteristics respectively. In addition, the extracted HSI and LiDAR features are fused through some operations to obtain the feature information more in line with the real situation. Finally, the above three data are fed into different branches of the DNL module, respectively. Extensive experiments on Houston dataset show that the proposed network is superior and more effective compared to several of the most advanced baselines in HSI and LiDAR joint classification missions.
BibTeX
@inproceedings{liu2021disentangled,
  author = {Wenxia Liu and Feng Gao and Junyu Dong},
  title = {{Disentangled Non-Local Network for Hyperspectral and LiDAR Data Classification}},
  booktitle = {IGARSS},
  year = {2021},
  doi = {10.1109/igarss47720.2021.9553313},
}
Figure