OGP-Net: Optical Guidance Meets Pixel-Level Contrastive Distillation for Robust Multi-Modal and Missing Modality Segmentation

Aniruddh Sikdar, Jayant Teotia, Suresh SundaramAAAI 2025 · DOI

Introduction

Semantic segmentation is a critical component for autonomous systems, yet standard RGB-based models often fail in challenging environments such as low-light or overexposed conditions. While Infrared (IR) sensors provide complementary thermal information, effectively fusing these disparate modalities remains difficult. Existing fusion techniques frequently encounter "negative transfer" due to the domain gap between light and heat, struggle with missing modalities (e.g., sensor failure at night), and often require massive computational resources that are unsuitable for edge deployment.

Methodology

To address these challenges, the authors propose OGP-Net (Optically-Guided Pixel-level contrastive learning Network). The framework utilizes a teacher-student distillation strategy to transfer high-quality semantic knowledge from a frozen RGB-only teacher (DeepLabV3+) to a multi-modal student network.

Core Components

  1. Distillation with Multi-View Contrastive (DMC): To bridge the RGB-IR domain gap, encoders share convolutional filters but maintain distinct Batch Normalization layers. A pixel-level contrastive loss ($L_{CL}$) is applied to align embeddings of the same semantic class across both modalities in a unified latent space.
  2. Distillation for Uni-modal Retention (DUR): This strategy ensures that the student's optical branch retains fine-grained textures from the RGB teacher, preventing IR data from degrading high-frequency details.
  3. Gated Spectral Unit (GSU): Rather than using static fusion methods like concatenation, the GSU dynamically learns the optimal weights for merging RGB and IR features, adapting to the strengths of each modality in real-time.
OGP-Net Architecture and Distillation Framework

Experimental Setup

OGP-Net was evaluated on three major multi-spectral benchmarks: MSRS (Multi-Spectral Road Scenarios), MVSS (Multi-modal Video Semantic Segmentation), and FMB (Full-day Multi-modal Benchmark). These datasets cover a wide range of urban scenes and lighting conditions, providing a rigorous test for both multi-modal fusion and missing modality robustness.

Key Results

OGP-Net demonstrates a significant leap in both accuracy and efficiency. On the MSRS dataset, it achieved a state-of-the-art 69.97 mIoU, outperforming heavy-weight competitors like CDDFuse and C.E.N. Notably, the model is highly efficient, utilizing only 14.24M parameters—roughly 14% of the parameters required by the C.E.N. model.

Quantitative Performance and Parameter Efficiency

In scenarios where the RGB modality is entirely missing (IR-only inference), OGP-Net maintains superior performance (66.77 mIoU on MSRS) compared to specialized missing-modality models like SKD-Net. This suggests that the contrastive distillation process successfully enriches the IR branch with semantic features typically reserved for optical sensors. Qualitative analysis further confirms that OGP-Net produces sharper boundaries for small objects and distant pedestrians compared to previous fusion methods.

Conclusion and Future Work

OGP-Net provides a robust and efficient solution for multi-modal semantic segmentation. By treating cross-modal alignment as a pixel-level contrastive learning problem, it achieves high accuracy with a lean architecture. Future research will focus on reducing the model's dependency on the pre-trained RGB teacher and integrating temporal dynamics to improve consistency in video-based segmentation tasks.

Summary updated

BibTeX
@inproceedings{sikdar2025ogp,
  author = {Aniruddh Sikdar and Jayant Teotia and Suresh Sundaram},
  title = {{OGP-Net: Optical Guidance Meets Pixel-Level Contrastive Distillation for Robust Multi-Modal and Missing Modality Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i7.32743},
}
Figure