Hyperspectral and Lidar Data Classification Based on Linear Self-Attention

Min Feng, Feng Gao, Jian Fang, Junyu Dong

Introduction

In remote sensing, the integration of Hyperspectral Images (HSI) and Light Detection and Ranging (LiDAR) data is a pivotal strategy for land-cover classification. HSI provides dense spectral information for material identification, while LiDAR offers critical 3D structural and elevation data. The primary challenge is the "heterogeneous gap"—the difficulty in effectively fusing data types representing fundamentally different physical properties. This paper introduces the Linear Self-Attention Fusion (LSAF) model, a novel framework designed to bridge this gap through adaptive cross-modal feature interaction.

Methodology

The LSAF model architecture is structured into three primary stages: feature extraction, linear self-attention-based interaction, and adaptive decision fusion.

The overall architecture of the proposed LSAF model, illustrating the feature extraction and fusion pipeline.

1. Feature Extraction Module

To ensure compatibility between modalities, the model employs a retrained HybridSN (a 3D-2D CNN) for HSI data and a series of three ConvBlocks for LiDAR data. This stage produces feature maps of consistent dimensions, facilitating subsequent fusion.

2. Linear Self-Attention (LiST) Module

The LiST module is the core innovation, acting as a "plug-and-play" component for cross-modal interaction. It integrates channel-wise attention, inspired by SENet, which recalibrates feature channels through a summation of HSI and LiDAR features. This is complemented by spatial attention using a softmax operator to prioritize significant spatial regions across the combined feature set.

3. Self-adaptive Decision Fusion

Rather than simple concatenation, the final classification output is a weighted summation of the HSI-specific result, the LiDAR-specific result, and the fused result. The weights are learned dynamically, allowing the model to prioritize the most reliable data source for each specific pixel.

Experimental Results

The LSAF model was evaluated using the Houston dataset, a standard benchmark in remote sensing consisting of 15 land-cover categories.

Comparison of classification performance (Overall Accuracy and class-wise accuracy) between LSAF and state-of-the-art methods.

As shown in the results table, LSAF achieved a state-of-the-art Overall Accuracy (OA) of 95.40%, outperforming existing methods such as $A^3CLNN$ (90.55%) and PToP CNN (92.48%). Notably, the model demonstrated significant improvements in challenging categories like "Healthy grass" (98.22%) and "Road" (95.50%), where spectral similarities often lead to misclassification in single-modality systems. Qualitatively, the classification maps produced by LSAF exhibit high visual fidelity to ground truth labels, with reduced noise in complex urban areas.

Conclusion

The LSAF model demonstrates that linear self-attention is a highly effective mechanism for bridging the heterogeneous gap between HSI and LiDAR data. By forcing cross-modal interaction and employing adaptive decision fusion, the model achieves superior classification accuracy. Future research will focus on evaluating the model's generalizability across diverse datasets and optimizing its computational efficiency for real-time applications.

Summary updated

BibTeX
@inproceedings{feng2021hyperspectral,
  author = {Min Feng and Feng Gao and Jian Fang and Junyu Dong},
  title = {{Hyperspectral and Lidar Data Classification Based on Linear Self-Attention}},
  booktitle = {IGARSS},
  year = {2021},
  doi = {10.1109/igarss47720.2021.9553769},
}
Figure