Learning Shared Cross-modality Representation Using Multispectral-LiDAR and Hyperspectral Data

Danfeng Hong, Jocelyn Chanussot, Naoto Yokoya, Jian Kang, Xiao Xiang Zhu

Abstract

Due to the ever-growing diversity of the data source, multi-modality feature learning has attracted more and more attention. However, most of these methods are designed by jointly learning feature representation from multi-modalities that exist in both training and test sets, yet they are less investigated in absence of certain modality in the test phase. To this end, in this letter, we propose to learn a shared feature space across multi-modalities in the training process. By this way, the out-of-sample from any of multi-modalities can be directly projected onto the learned space for a more effective cross-modality representation. More significantly, the shared space is regarded as a latent subspace in our proposed method, which connects the original multi-modal samples with label information to further improve the feature discrimination. Experiments are conducted on the multispectral-Lidar and hyperspectral dataset provided by the 2018 IEEE GRSS Data Fusion Contest to demonstrate the effectiveness and superiority of the proposed method in comparison with several popular baselines.

Introduction

In remote sensing, researchers often have access to diverse data sources, such as active sensors like LiDAR (providing elevation and structure) and passive sensors like Hyperspectral Imaging (HSI, providing rich chemical signatures). While combining these modalities typically improves results, most existing fusion methods assume all data types are available during both training and testing. In practice, hyperspectral data is often expensive to acquire over large areas, whereas Multispectral-LiDAR may be more readily available. This paper addresses the Cross-modality Learning (CML) challenge: using a small, overlapping set of HSI and LiDAR data to train a model that performs effectively on larger areas where only LiDAR data exists.

Methodology

The authors introduce $\ell_1$-CoSpace, a framework that learns a shared latent subspace between Multispectral-LiDAR and Hyperspectral data. The approach consists of three main components:

  1. Shared Subspace Learning: The model learns modality-specific projections that map both HSI and LiDAR data into a single, shared latent space, allowing different data types to be represented in a common format.
  2. Graph-Based Alignment: A joint Laplacian matrix is used to ensure that samples from the same class remain close together in the latent space, preserving the local structure of the data regardless of the original modality.
  3. Sparse Regression ($\ell_1$-norm): By employing sparse regression instead of standard ridge regression ($\ell_2$), the model performs automatic feature selection. This identifies the most discriminative features in the shared space, making the model more robust to noise and improving generalization.
The proposed framework for cross-modality learning, showing the training (TR) and testing (TE) phases.

Experimental Setup

The study utilizes data from the 2018 IEEE GRSS Data Fusion Contest (DFC2018). The dataset includes Multispectral-LiDAR (acquired at 1550 nm, 1064 nm, and 532 nm) and Hyperspectral Images (covering 48 spectral bands). To ensure pixel-to-pixel alignment, the LiDAR data was downsampled to match the 1-meter spatial resolution of the HSI.

Key Results and Discussion

The $\ell_1$-CoSpace method was compared against several baselines using two classifiers: Linear SVM (LSVM) and Canonical Correlation Forests (CCF).

Quantitative comparison of classification performance across different algorithms and classifiers.

Key findings include:

  • Overall Accuracy (OA): Using the CCF classifier, $\ell_1$-CoSpace achieved an OA of 68.17%, a significant improvement over the 59.79% baseline (using only LiDAR data).
  • Superiority of Sparsity: The sparse version ($\ell_1$) consistently outperformed the original $\ell_2$-CoSpace (which achieved 66.25% OA), proving that feature selection within the shared subspace is beneficial.
  • Class-Specific Gains: The model showed dramatic improvements in difficult-to-distinguish classes, such as specific vegetation types and urban structures, where the "knowledge" transferred from HSI data during training was most valuable.

Conclusion

$\ell_1$-CoSpace successfully bridges the modality gap in remote sensing by learning a shared, sparse representation. By forcing the regression matrix to be sparse, the model effectively ignores redundant dimensions and focuses on features that distinguish land-cover classes. Future work will investigate non-linear mappings, such as deep neural networks, to handle even more complex and heterogeneous data sources.

Summary updated

Abstract

Due to the ever-growing diversity of the data source, multi-modality feature learning has attracted more and more attention. However, most of these methods are designed by jointly learning feature representation from multi-modalities that exist in both training and test sets, yet they are less investigated in absence of certain modality in the test phase. To this end, in this letter, we propose to learn a shared feature space across multi-modalities in the training process. By this way, the out-of-sample from any of multi-modalities can be directly projected onto the learned space for a more effective cross-modality representation. More significantly, the shared space is regarded as a latent subspace in our proposed method, which connects the original multi-modal samples with label information to further improve the feature discrimination. Experiments are conducted on the multispectral-Lidar and hyperspectral dataset provided by the 2018 IEEE GRSS Data Fusion Contest to demonstrate the effectiveness and superiority of the proposed method in comparison with several popular baselines.
BibTeX
@article{hong2020learning,
  author = {Danfeng Hong and Jocelyn Chanussot and Naoto Yokoya and Jian Kang and Xiao Xiang Zhu},
  title = {{Learning Shared Cross-modality Representation Using Multispectral-LiDAR and Hyperspectral Data}},
  journal = {IEEE Geosci. Remote. Sens. Lett.},
  year = {2020},
  url = {https://openreview.net/forum?id=Giu9Q1Tin9},
}
Figure