An Alternative of LIDAR in Nighttime: Unsupervised Depth Estimation Based on Single Thermal Image

Yawen Lu, Guoyu LuWACV 2021

TL;DR

This paper introduces a novel unsupervised method for estimating depth from single thermal images, overcoming the limitations of traditional depth sensors like LiDAR in low-light conditions. By leveraging visible light cameras for training and employing a unique image translation technique, the approach achieves accurate depth estimation without requiring ground truth depth data.

What Problem Does This Paper Address?

Accurate depth estimation is crucial for autonomous systems like self-driving cars and robots. While LiDAR sensors are commonly used, they have limitations: low resolution and the potential to harm people with their laser emissions. Existing image-based depth estimation methods primarily rely on RGB cameras, which perform poorly in dark or nighttime environments due to insufficient lighting. Thermal cameras offer a solution by capturing heat signatures, but they lack the rich texture information found in RGB images, making traditional depth estimation challenging. Furthermore, acquiring ground truth depth data for training supervised models is expensive and labor-intensive. This paper aims to bridge this gap by enabling robust, unsupervised depth estimation from single thermal images, even in challenging low-light scenarios.

Methods

The core of the proposed method lies in an unsupervised framework that leverages visible light cameras to facilitate the training of a thermal image depth estimation model. The key components are:

  • Image Translation: A Generative Adversarial Network (GAN), specifically a dual CycleGAN, is employed to translate visible color images into synthetic thermal images. This allows the model to learn correspondences between the two modalities, even when paired data is not available. The translation is guided by cycle consistency, adversarial loss, and novel illumination consistency and contrast losses to ensure the synthetic thermal images are realistic.
  • Disparity Initialization: Since thermal images are texture-poor, the method uses stereo color cameras to obtain an initial disparity map. This map, which is physically close to the thermal camera's view, serves as an initialization for the thermal image depth estimation.
  • Depth Estimation Network: A CNN-based encoder-decoder architecture is used to refine the initial disparity map. This network is trained using a combination of loss functions, including:
  • Appearance Matching Loss: Ensures reconstructed images match the original input.
  • Edge-aware Smoothness Loss: Prevents unrealistic depth variations in occluded or low-texture regions.
  • Illuminance Loss (Llocal_lx): A novel loss that considers correlation, average illuminance, and contrast to improve the quality of generated thermal images.

During inference, only a single thermal image is required to generate the depth map.

Datasets

  • Collected Dataset: A rectified multi-spectral stereo dataset was collected, containing 10,000 image pairs covering various indoor and outdoor scenes. This dataset was used for training and evaluation, with ground truth depth maps generated from stereo reconstruction of thermal images.
  • KITTI Dataset: Used for evaluating the model's performance on autonomous driving scenarios. The model was adapted to translate KITTI's color images to a synthetic thermal domain for testing.

Key Insights and Ideas

  • Bridging Modalities: The successful translation of visible images to thermal images is a critical enabler for depth estimation in low-light conditions, allowing the use of readily available color camera data for training.
  • Unsupervised Learning for Thermal Depth: The framework demonstrates that robust depth estimation from thermal images can be achieved without requiring expensive ground truth depth labels, making it more practical for real-world applications.
  • Leveraging Color for Initialization: Using stereo color cameras to provide an initial disparity map for thermal images significantly improves the accuracy and stability of the subsequent depth estimation process.
  • Novel Loss Functions: The introduction of illumination consistency and contrast losses for image translation, and the local illuminance loss for depth estimation, are key to handling the unique characteristics of thermal imagery.

Key Results

  • The proposed method achieves state-of-the-art performance on the collected dataset, outperforming existing monocular depth estimation techniques across various error and accuracy metrics (e.g., Abs Rel, Sq Rel, RMSE, RMSE log).
  • Qualitative results show that the model generates superior depth maps, effectively handling challenging scenarios and producing clearer object boundaries compared to other methods.
  • The image translation component also demonstrates strong performance, achieving the best results in PSNR, SSIM, COS similarity, and RMSE compared to other state-of-the-art translation methods.

Limitations and Future Work

  • Initialization Dependency: While the method is unsupervised for the thermal depth estimation itself, it relies on stereo color cameras for initial disparity estimation. The performance might be affected if the stereo color reconstruction is poor.
  • Domain Gap: Although image translation is used, there might still be subtle differences between translated and real thermal images, which could impact the final depth accuracy.
  • Future Work: The authors suggest extending this framework to other computer vision tasks like Structure from Motion (SfM), SLAM, and object detection, particularly for applications relying on passive illumination sensors.

Summary updated

BibTeX
@inproceedings{lu2021alternative,
  author = {Yawen Lu and Guoyu Lu},
  title = {{An Alternative of LIDAR in Nighttime: Unsupervised Depth Estimation Based on Single Thermal Image}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure