Real-Time Restoration of Dark Stereo Images

Mohit Lamba, M. V. A. Suhas Kumar, Kaushik MitraWACV 2023

TL;DR

This paper introduces a novel, lightweight hybrid U-Net architecture for real-time restoration of extremely dark stereo images. By processing stereo views at different scales and employing a differentiable depth-aware loss module, it achieves state-of-the-art visual enhancement and preserves epipolar geometry, outperforming existing methods in both speed and efficiency.

What Problem Does This Paper Address?

While low-light image enhancement has seen significant progress, the specific challenge of restoring dark stereo images has been largely overlooked. This gap is critical for enabling night-time capabilities in applications that rely on 3D information, such as autonomous driving (where LiDAR is costly) and augmented/virtual reality. Existing monocular methods fail to leverage stereo information, and current stereo models are often optimized for depth estimation, not visual enhancement, and typically rely on computationally expensive 3D convolutions. Furthermore, many existing stereo models are trained on specific datasets like KITTI or SceneFlow, limiting their applicability to arbitrary stereo setups.

Methods

The core of the proposed solution is a hybrid U-Net architecture designed for speed and efficiency. Key aspects include:

  • Scale-Space Processing: The network processes stereo features at multiple resolutions. In the initial, coarser scales where disparity is large, left and right features are processed independently to handle misalignment. As the scale-space becomes finer, disparity decreases, and the network jointly processes features to improve efficiency and preserve epipolar geometry.
  • 2D Convolutions Only: To ensure real-time performance, the architecture strictly avoids computationally intensive 3D convolutions and attention modules.
  • Depth-Aware Loss Module: To enforce epipolar constraints without relying on pre-trained, computationally expensive depth estimation models, a novel differentiable loss module is introduced. This module uses classical computer vision techniques to compute coarse depth estimates from the enhanced stereo views. It then enforces geometric consistency by comparing these estimates with ground-truth disparities (obtained from GT stereo views) through an L1 loss. This module is designed to be lightweight and can be used with any stereo setup by simply adjusting a single hyperparameter, dispmax.

Datasets

The paper uses the following datasets:

  • KITTI 2015: Well-lit stereo images are transformed into low-light images following a principled approach (detailed in the supplementary) to create a custom dataset for training and testing. Ground truth depth maps from LiDAR are used.
  • CityScape: Similar to KITTI, well-lit images are converted to low-light. Ground truth depth is obtained using SGM (Semi-Global Matching).
  • L3F Light Field dataset: Used for qualitative evaluation on real low-light stereo images.

Key Insights and Ideas

  • Stereo Alignment at Coarser Scales: The observation that stereo feature alignment improves at coarser scales is crucial. This allows for a hybrid processing strategy that balances individual view processing with joint processing for efficiency.
  • Differentiable Depth-Aware Loss for Epipolar Constraints: Instead of relying on external, potentially slow, depth estimators, a differentiable loss module that leverages classical vision techniques offers a computationally efficient way to enforce geometric constraints during training. This is a significant departure from typical stereo pipelines.
  • Lightweight Architecture for Real-World Deployment: The explicit focus on avoiding 3D convolutions and attention mechanisms, while still achieving good results, highlights the importance of computational efficiency for practical applications.
  • Generalizability of the Loss Module: The fact that the Depth-Aware Loss Module can be used with arbitrary stereo setups by tuning dispmax makes it highly versatile.

Key Results

  • Quantitative Performance: The proposed method achieves state-of-the-art visual enhancement (PSNR, SSIM) and depth estimation metrics (RMSE, D1%) on the KITTI dataset. On CityScape, it also outperforms other methods while maintaining a fast inference speed.
  • Speedup: The method offers a significant speedup of 4-60× compared to existing methods.
  • Computational Efficiency: It uses 15-100× lower floating-point operations than computationally expensive methods.
  • Qualitative Results: Visual results show that the restored images are nearly identical to ground truth, and downstream tasks like depth estimation and semantic segmentation perform comparably to those on GT images. The method also effectively suppresses noise and enhances color in extremely dark conditions.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the Depth-Aware Loss Module provides good coarse-level disparity for training, it is not intended to replace state-of-the-art stereo depth models for precise depth estimation. The method also notes that for extremely low-light images, recovering very fine textures can be challenging.
  • Future Work: The authors suggest exploring parameterizing the scale at which feature merging occurs as a function of stereo baseline, although this might increase training complexity.

Summary updated

BibTeX
@inproceedings{lamba2023real,
  author = {Mohit Lamba and M. V. A. Suhas Kumar and Kaushik Mitra},
  title = {{Real-Time Restoration of Dark Stereo Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure