Abstract
Due to the ever-growing diversity of the data source, multimodality feature learning has attracted more and more attention. However, most of these methods are designed by jointly learning feature representation from multimodalities that exist in both training and test sets, yet they are less investigated in the absence of certain modality in the test phase. To this end, in this letter, we propose to learn a shared feature space across multimodalities in the training process. By this way, the out-of-sample from any of multimodalities can be directly projected onto the learned space for a more effective cross-modality representation. More significantly, the shared space is regarded as a latent subspace in our proposed method, which connects the original multimodal samples with label information to further improve the feature discrimination. Experiments are conducted on the multispectral-Light Detection and Ranging (LIDAR) and hyperspectral data set provided by the 2018 IEEE GRSS Data Fusion Contest to demonstrate the effectiveness and superiority of the proposed method in comparison with several popular baselines.
Introduction
In remote sensing, multi-modal data fusion—combining active sensors like LiDAR (elevation) with passive sensors like Hyperspectral imaging (HSI, material composition)—is a standard approach for improving classification accuracy. However, a significant practical hurdle exists: high-quality HSI data is often expensive and limited in coverage, while Multispectral (MS) and LiDAR data are more widely available.
Most existing fusion techniques require all modalities to be present during both training and testing. This paper addresses the "missing modality" problem: leveraging a small amount of overlapping HSI data during training to "teach" a model to better classify wide-scale MS-LiDAR data, even when the HSI data is unavailable during the test phase.
Methodology
The proposed framework builds upon Common Subspace Learning (CoSpace), aiming to project disparate modalities into a shared latent feature space. The HSI data acts as a "teacher," capturing discriminative spectral power that is then embedded into the projection matrix for the MS-LiDAR data.
Key Technical Contributions:
- $\ell_1$-CoSpace: The authors introduce sparse regression ($\ell_1$-norm) to connect the latent subspace to class labels. This serves as an automatic feature selector, enhancing robustness against noise compared to traditional ridge regression ($\ell_2$-norm).
- Graph-Based Alignment: A joint Laplacian matrix is utilized to enforce manifold alignment, ensuring that points similar in their original modality remain proximal in the shared latent space.
- Optimization via ADMM: The model utilizes the Alternating Direction Method of Multipliers to solve the complex, non-convex objective function efficiently.
Experimental Results
The study utilized the 2018 IEEE GRSS Data Fusion Contest (DFC2018) dataset, featuring 48-band HSI and 9-band MS-LiDAR data aligned to a 1-meter Ground Sampling Distance.
Results demonstrate that the proposed $\ell_1$-CoSpace significantly outperforms standard baselines:
- Accuracy Improvement: Using a Canonical Correlation Forest (CCF) classifier, the method achieved an Overall Accuracy (OA) of 68.17%, an 8.38% absolute improvement over the baseline MS-LiDAR accuracy of 59.79%.
- Sparsity Benefits: The $\ell_1$ version outperformed the $\ell_2$-CoSpace by nearly 2% in OA, validating the effectiveness of sparse modeling in filtering redundant features.
- Class Gains: Notable improvements were observed in challenging categories such as "Trains," "Stadium Seats," and specific vegetation types, where spectral knowledge transfer was most impactful.
Conclusion
This research successfully demonstrates that a shared latent space can bridge the gap between heterogeneous sensors. By treating HSI as a temporary training aid, the model effectively transfers discriminative knowledge to more widely available MS-LiDAR data. Future work will focus on addressing spatial alignment sensitivities and exploring deep learning architectures to further enhance cross-modality representation.
Summary updated