Shared Manifold Learning Using a Triplet Network for Multiple Sensor Translation and Fusion with Missing Data

Aditya Dutt, Alina Zare, Paul D. Gader

Abstract

Heterogeneous data fusion is essential for enhancing the robustness and accuracy of remote sensing algorithms. However, aligning disparate modalities—such as hyperspectral imagery (HSI), LiDAR, and Synthetic Aperture Radar (SAR)—into a cohesive representation remains a significant challenge. This paper introduces the Contrastive learning based MultiModal Alignment Network (CoMMANet), a framework designed to align multi-sensor data into a shared, discriminative manifold. By utilizing a multimodal triplet autoencoder, CoMMANet clusters latent space representations such that samples of the same class across different modalities are mapped in close proximity. This shared manifold enables a unified classification framework and facilitates efficient sensor translation in scenarios where data is missing. Experimental results on the MUUFL Gulfport and Berlin datasets demonstrate state-of-the-art performance, achieving mean overall accuracies of 94.3% and 71.26%, respectively.

Introduction

In the field of remote sensing, multiple sensors provide complementary views of the Earth's surface. Hyperspectral sensors offer detailed chemical and spectral information, LiDAR provides structural height data, and SAR can penetrate cloud cover. While fusing these sources is theoretically powerful, their fundamentally different data structures make alignment difficult. Traditional methods often rely on simple feature concatenation, which fails to account for the complex relationships between modalities and breaks down when a sensor is missing.

CoMMANet addresses this "multimodal alignment" challenge by creating a compact, shared mathematical space—a manifold—where the identity of an object (e.g., a "road" or a "tree") is preserved regardless of the sensor used to observe it.

The CoMMANet Framework

The core of the proposed architecture is a deep learning framework that combines autoencoders with contrastive learning. The system is built upon four primary innovations:

1. Multimodal Triplet Autoencoders

Standard autoencoders compress high-dimensional sensor data into a low-dimensional latent space. CoMMANet enhances this by incorporating a triplet loss function. This technique, originally popularized in facial recognition, ensures that the network maps samples of the same class (anchors and positives) close together while pushing different classes (negatives) apart in the latent space.

2. Inter-sensor Triplet Loss

To ensure the sensors "speak the same language," the authors introduced an inter-sensor triplet loss. In this setup, an anchor from one sensor (e.g., HSI) is compared against positives and negatives from another sensor (e.g., LiDAR). This forces the network to align the internal representations of different modalities into a single coordinate system.

3. Similarity Enhancement (SE) Term

A key finding in the research was that standard triplet loss alone often left a small gap between the clusters of different sensors. The Similarity Enhancement (SE) term acts as the "glue" for the manifold, specifically minimizing the distance between the anchors of different modalities to ensure near-perfect overlap in the latent space.

4. Latent Space Sensor Translation

Because the sensors share a manifold, the system can perform sensor translation. If a sensor is missing during inference, a simple regression network can predict the missing sensor's latent representation using the available data. This allows the system to "hallucinate" structural information from spectral data or vice versa.

Experimental Results

The model was validated across three diverse datasets to test its versatility:

  • AVIRIS-NG and NEON: A dataset featuring hyperspectral data from two platforms and LiDAR data over Healy, Alaska.
  • MUUFL Gulfport: A benchmark dataset of co-registered HSI and LiDAR over a university campus.
  • HS-SAR Berlin: A challenging dataset pairing simulated EnMAP hyperspectral data with Sentinel-1 SAR data.

Performance Metrics

CoMMANet demonstrated superior performance compared to existing state-of-the-art (SOTA) methods, including Vision Transformers and specialized fusion networks. On the MUUFL dataset, the model achieved a mean overall accuracy of 94.3%. On the more complex Berlin dataset (SAR-HSI fusion), it reached 71.26%, outperforming previous benchmarks like CoSpace and CCRNet.

Furthermore, the sensor translation capabilities were highly effective. The model reconstructed missing LiDAR height and intensity maps from HSI data with remarkably low Mean Squared Error (MSE), proving that the shared manifold successfully captures the underlying physical characteristics of the scene.

Discussion and Insights

The research highlighted several critical factors for successful multimodal learning:

  • The Importance of Semi-Hard Mining: The authors found that using "hard" triplets (the most difficult examples) too early in training caused model collapse. Utilizing "semi-hard" triplets provided the stability necessary for the network to converge on a robust manifold.
  • Efficiency of Latent Translation: By performing translation in the low-dimensional latent space rather than the high-dimensional image space, the model remains computationally efficient while maintaining high accuracy.
  • Sensor-Agnostic Classification: Once the manifold is established, a classifier trained on one modality can effectively classify data from another, though the best results are consistently achieved by fusing the shared embeddings.

Conclusion

CoMMANet represents a significant step forward in heterogeneous data fusion for remote sensing. By aligning sensors into a shared manifold using contrastive learning and similarity enhancement, the framework provides a robust solution for both classification and missing data recovery. Future work will explore the integration of attention mechanisms to dynamically weigh sensor contributions and improve generalization across even larger, more diverse geographic areas.

Summary updated

Abstract

Heterogeneous data fusion can enhance the robustness and accuracy of an algorithm on a given task. However, due to the difference in various modalities, aligning the sensors and embedding their information into discriminative and compact representations is challenging. In this paper, we propose a Contrastive learning based MultiModal Alignment Network (CoMMANet) to align data from different sensors into a shared and discriminative manifold where class information is preserved. The proposed architecture uses a multimodal triplet autoencoder to cluster the latent space in such a way that samples of the same classes from each heterogeneous modality are mapped close to each other. Since all the modalities exist in a shared manifold, a unified classification framework is proposed. The resulting latent space representations are fused to perform more robust and accurate classification. In a missing sensor scenario, the latent space of one sensor is easily and efficiently predicted using another sensor's latent space, thereby allowing sensor translation. We conducted extensive experiments on a manually labeled multimodal dataset containing hyperspectral data from AVIRIS-NG and NEON, and LiDAR (light detection and ranging) data from NEON. Lastly, the model is validated on two benchmark datasets: Berlin Dataset (hyperspectral and synthetic aperture radar) and MUUFL Gulfport Dataset (hyperspectral and LiDAR). A comparison made with other methods demonstrates the superiority of this method. We achieved a mean overall accuracy of 94.3% on the MUUFL dataset and the best overall accuracy of 71.26% on the Berlin dataset, which is better than other state-of-the-art approaches.
BibTeX
@article{dutt2022shared,
  author = {Aditya Dutt and Alina Zare and Paul D. Gader},
  title = {{Shared Manifold Learning Using a Triplet Network for Multiple Sensor Translation and Fusion with Missing Data}},
  journal = {IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens.},
  year = {2022},
  doi = {10.1109/jstars.2022.3217485},
}
Figure