Twilight SLAM: Navigating Low-Light Environments

Surya Pratap Singh, Billy Mazotti, Sarvesh Mayilvahanan, Guoyuan Li, Dhyey Manish Rajani, Maani Ghaffari

TL;DR

"Twilight SLAM" investigates whether state-of-the-art image enhancement can rescue Visual SLAM (Simultaneous Localization and Mapping) in near-total darkness. By benchmarking various "digital flashlights" (enhancement algorithms) against standard SLAM frameworks, the researchers found that while enhancement significantly boosts accuracy in semi-darkness, extremely dark environments remain a formidable frontier where even the best algorithms struggle.

What Problem Does This Paper Address?

Visual SLAM (VSLAM) is the "eyes" of an autonomous robot, allowing it to map an environment and locate itself using only camera data. However, VSLAM has a major Achilles' heel: it requires light. In low-light environments—like warehouses at night, tunnels, or hospitals with dimmed lights—traditional feature extractors (the algorithms that find "landmarks" in an image) fail to find enough points to track, causing the robot to lose its way.

While previous research has looked at "dim" lighting, this paper pushes into the "twilight" zone, specifically categorizing and testing "semi-dark" and "dark" environments. The authors ask a fundamental question: Can we simply use deep-learning-based image enhancement as a pre-processing step to make standard SLAM work in the dark?

Methods

The researchers developed a "plug-and-play" pipeline to evaluate combinations of image enhancers and SLAM frameworks.

The Enhancers (The "Digital Flashlights"): * EnlightenGAN: An unsupervised generative adversarial network that brightens images without needing "perfectly lit" training pairs. * Bread: A model that separately handles color and noise, aiming for more realistic results. * Zero-DCE: A lightweight method that treats enhancement as a curve-estimation problem, making it very fast. * Dual: An exposure correction method that estimates both forward and reverse illumination.

The SLAM Frameworks: * ORB-SLAM3: The industry standard, which uses hand-crafted geometric features (ORB). * SuperPoint-SLAM: A more modern approach that uses deep-learning-based feature detection and description.

The team established a systematic framework for categorizing light levels based on normalized mean intensity:

  • Dark: $\le 0.15$
  • Semi-dark: $0.15$ to $0.28$
  • Mild-daylight: $0.28$ to $0.35$

Datasets

The study utilized two primary sources:

  1. KITTI: The gold standard for autonomous driving datasets, used here to represent "mild-daylight" scenarios (sequences 04, 06, and 07).
  2. ETH3D: Specifically the sfm_house_loop (semi-dark) and sfm_lab_room2 (dark) sequences. These were captured with handheld stereo cameras in cluttered, poorly lit indoor and outdoor environments. Since motion capture wasn't available for these, the authors used Structure-from-Motion to generate ground truth data.

Key Insights and Ideas

  • EnlightenGAN is the MVP: Across almost all dark and semi-dark tests, EnlightenGAN emerged as the most robust enhancer. Because it is trained unsupervised, it generalizes better to the "messy" lighting of real-world scenes compared to models trained on synthetic data.
  • The "Semi-Dark" Sweet Spot: Image enhancement provides a massive boost in semi-dark environments. It allows the SLAM system to find significantly more features, leading to much lower localization errors.
  • The Darkness Ceiling: In "Dark" environments (intensity $\le 0.15$), the improvements are only incremental. If the original image is too dark, the enhancer often introduces "hallucinated" noise that can confuse feature detectors just as much as the darkness itself.
  • ORB-SLAM3 vs. SuperPoint: Interestingly, ORB-SLAM3 (the older, geometric method) paired with EnlightenGAN often outperformed the newer SuperPoint-SLAM in the darkest scenarios, suggesting that traditional geometric descriptors are still remarkably resilient when given a "brightened" input.

Key Results

The researchers measured performance using NCPP (number of compared pose pairs—essentially how many landmarks the robot successfully tracked) and RMSE (Root Mean Square Error in meters).

  • Accuracy Boost: In the semi-dark sfm_house sequence, ORB-SLAM3 with EnlightenGAN reduced the localization error (RMSE) from 0.43m to 0.15m—a nearly 3x improvement.
  • Feature Extraction: In the dark sfm_lab sequence, the number of tracked features (NCPP) jumped from 63 to 90 when using the Dual enhancer with ORB-SLAM3.
  • Best Combo: The paper concludes that EnlightenGAN + ORB-SLAM3 is the most reliable combination for navigating challenging low-light environments.

Limitations and Future Work

The authors acknowledge that while enhancement helps, it isn't a silver bullet for total darkness. Extremely dark scenes still suffer from "feature detection failures" where the SLAM system simply cannot find enough reliable points to stay localized.

Future directions include: * Investigating why ORB features sometimes perform better in "dark-enhanced" images than in "semi-dark" ones (an anomaly noted in their data). * Integrating more advanced illumination map interpolation techniques (like LIME). * Optimizing the computational overhead, as some learning-based SLAM methods (like SuperPoint) are significantly slower than traditional ones.

Summary updated

Abstract

This paper presents a detailed examination of low-light visual Simultaneous Localization and Mapping (SLAM) pipelines, focusing on the integration of state-of-the-art (SOTA) low-light image enhancement algorithms with standard and contemporary SLAM frameworks. The primary objective of our work is to address a pivotal question: Does illuminating visual input significantly improve localization accuracy in both semi-dark and dark environments? In contrast to previous works that primarily address partially dim-lit datasets, we comprehensively evaluate various low-light SLAM pipelines across obscurely-lit environments. Employing a meticulous experimental approach, we qualitatively and quantitatively assess different combinations of image enhancers and SLAM frameworks, identifying the best-performing combinations for feature-based visual SLAM. The findings advance low-light SLAM by highlighting the practical implications of enhancing visual input for improved localization accuracy in challenging lighting conditions. This paper also offers valuable insights, encouraging further exploration of visual enhancement strategies for enhanced SLAM performance in real-world scenarios.
BibTeX
@misc{singh2023twilight,
  author = {Surya Pratap Singh and Billy Mazotti and Sarvesh Mayilvahanan and Guoyuan Li and Dhyey Manish Rajani and Maani Ghaffari},
  title = {{Twilight SLAM: Navigating Low-Light Environments}},
  year = {2023},
  eprint = {2304.11310},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.11310},
}
Figure