Multispectral Transfer Network: Unsupervised Depth Estimation for All-Day Vision

Namil Kim, Yukyung Choi, Soonmin Hwang, In So KweonAAAI 2018 · OpenReview

Abstract

To achieve true all-day vision for autonomous systems, perception must remain robust in conditions where standard RGB sensors fail, such as total darkness. This paper introduces the Multispectral Transfer Network (MTN), a novel unsupervised framework designed to estimate pixel-wise depth from a single thermal image. By leveraging geometric priors and chromaticity clues through a multi-task learning approach, MTN bridges the domain gap between thermal and visible spectrums. The architecture introduces a specialized "Interleaver" module to share features efficiently without direct layer sharing. Experimental results on a new multispectral stereo dataset demonstrate that MTN significantly outperforms existing baselines, particularly in challenging nighttime scenarios.

The Challenge of All-Day Perception

Standard depth estimation techniques are heavily reliant on RGB cameras, which are susceptible to failure in low-light environments like tunnels or nighttime streets. While active sensors like LiDAR can operate in the dark, they are often cost-prohibitive and provide sparse data. Thermal (Long-Wavelength Infrared) sensors offer a compelling alternative due to their lighting invariance, but they face two primary challenges: a lack of large-scale annotated depth datasets and a significant domain gap where thermal images lack the fine-grained texture and color information typically used for monocular depth inference.

Proposed Method: Multispectral Transfer Network (MTN)

The MTN framework addresses these challenges by using an unsupervised training strategy. During training, the model uses a calibrated RGB stereo pair to provide a geometric signal. The network learns to predict a disparity map from a thermal image, which is then used to warp a right-view RGB image to reconstruct the left-view. The reconstruction error serves as the training signal, eliminating the need for manual depth labels.

Key Architectural Innovations

  • The Interleaver Module: A core component of the MTN is the Interleaver block. Inspired by gating mechanisms in recurrent neural networks, it uses global average pooling to intelligently share feature information between the primary depth estimation task and a secondary chromaticity (color) prediction task. This allows the model to learn richer geometric structures common to both domains.
  • Multi-task Learning: By forcing the network to reconstruct the chromaticity (Cb and Cr channels) of the scene alongside depth, the encoder is incentivized to capture more detailed scene layouts.
  • Adaptive Scaled Sigmoid ($S_{ass}$): To stabilize training and accurately represent objects close to the camera, the authors implemented a sigmoid function that adaptively scales its range as training progresses.
The proposed MTN architecture (a), the Interleaver component (b), and comparisons with baseline multi-task architectures (c).

Datasets and Evaluation

The researchers introduced a Multispectral Stereo Dataset containing 7,383 sets of stereo RGB and thermal images captured across campus, residential, and urban environments. For evaluation, 3D LiDAR data was used as ground truth. To handle thermal contrast variations caused by heat shifts throughout the day, a data-driven photometric correction method was applied to normalize the thermal features.

Key Results and Insights

MTN demonstrates superior performance and generalization across various driving conditions:

  • Accuracy Gains: In daytime tests, MTN improved the Root Mean Square Error (RMSE) by 22% at a 50m range compared to single-task networks.
  • Nighttime Superiority: In total darkness, MTN's accuracy was three times higher than RGB-based methods, which fail entirely without external illumination.
  • Visual Fidelity: Qualitatively, the depth maps produced by MTN are sharper and preserve object boundaries—such as cars and trees—more effectively than previous multi-task architectures like "Deep-shared" (DsMTN) or "Low-shared" (LsMTN) models.
Visual comparison of thermal input and the resulting depth estimation across different lighting and environmental conditions.

Conclusion

The Multispectral Transfer Network provides a robust solution for all-day depth estimation by successfully transferring knowledge from the visible to the thermal domain. By utilizing an unsupervised multi-task framework and the novel Interleaver module, the system achieves high-resolution depth maps without the need for expensive LiDAR supervision. Future work will focus on refining loss functions to further reduce blurriness and improve long-range depth accuracy.

Summary updated

Abstract

To understand the real-world, it is essential to perceive in all-day conditions including cases which are not suitable for RGB sensors, especially at night. Beyond these limitations, the innovation introduced here is a multispectral solution in the form of depth estimation from a thermal sensor without an additional depth sensor.Based on an analysis of multispectral properties and the relevance to depth predictions, we propose an efficient and novel multi-task framework called the Multispectral Transfer Network (MTN) to estimate a depth image from a single thermal image. By exploiting geometric priors and chromaticity clues, our model can generate a pixel-wise depth image in an unsupervised manner. Moreover, we propose a new type of multitask module called Interleaver as a means of incorporating the chromaticity and fine details of skip-connections into the depth estimation framework without sharing feature layers. Lastly, we explain a novel technical means of stably training and covering large disparities and extending thermal images to data-driven methods for all-day conditions. In experiments, we demonstrate the better performance and generalization of depth estimation through the proposed multispectral stereo dataset, including various driving conditions.
BibTeX
@inproceedings{kim2018multispectral,
  author = {Namil Kim and Yukyung Choi and Soonmin Hwang and In So Kweon},
  title = {{Multispectral Transfer Network: Unsupervised Depth Estimation for All-Day Vision}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=ByN-p0g_ZB},
}
Figure