Improving Nighttime Retrieval-Based Localization

Hugo Germain, Guillaume Bourmaud, Vincent LepetitarXiv 2018 · DOI · arXiv

Abstract

Outdoor visual localization is a crucial component of many computer vision systems, including autonomous driving and augmented reality. We propose an approach to localization from images that is designed to explicitly handle the strong variations in appearance happening between daytime and nighttime. As revealed by recent long-term localization benchmarks, both traditional feature-based and retrieval-based approaches still struggle to handle such changes. Our novel localization method combines a state-of-the-art image retrieval architecture with condition-specific sub-networks, allowing the computation of global image descriptors that are explicitly dependent on the capturing conditions. We show that our approach improves localization by a factor of almost 300% compared to popular VLAD-based methods on nighttime localization.

Introduction

Visual localization—the process of determining a camera's 6 Degree-of-Freedom (6 DoF) pose from an image—is a fundamental challenge in robotics and computer vision. While significant progress has been made in "fair-weather" scenarios, real-world applications require systems that function reliably across drastic environmental changes. Most existing localization databases are constructed from images captured during clear daylight. When a system attempts to localize a query image taken at night against such a database, it often fails due to the massive appearance gap.

The Challenge: Feature Failure and the Domain Gap

The difficulty of nighttime localization stems from two primary issues:

  1. Local Feature Failure: Traditional "feature-based" methods rely on matching local points of interest (like SIFT or ORB). At night, these points often disappear, are obscured by shadows, or change so drastically that reliable matching becomes impossible.
  2. The Domain Gap: "Retrieval-based" methods treat localization as an image-search problem, finding the most similar image in a database. However, standard neural networks struggle to recognize that a nighttime street corner is the same physical location as its daytime counterpart because the pixel-level distributions are fundamentally different.

Condition-Aware Retrieval Architecture

To bridge this gap, we propose a Siamese network architecture that is "aware" of the capturing conditions. Our approach utilizes three key innovations:

Condition-Specific "Expert" Sub-networks

Instead of a single monolithic backbone, our network features several parallel branches ($f_c$). During both training and inference, an image is routed through a specific branch based on its condition (e.g., a "Night" branch or a "Sun" branch). This allows the network to learn specialized weights for different environments without diluting the features. Crucially, since only one branch is active for any given image, there is zero additional computational overhead during inference compared to a standard network.

Condition-Agnostic Blocks

After initial specialized processing, features pass through shared blocks ($g$). These blocks ensure that features from different conditions are mapped into a common representation space, allowing a night query to be successfully matched to a day reference.

Generalized-Mean (GeM) Pooling

We move away from traditional NetVLAD layers in favor of Generalized-Mean (GeM) pooling. GeM acts as a tunable middle ground between average and max pooling, providing a more discriminative global descriptor that is robust to the lighting variations typical of nighttime scenes.

Experimental Results

We evaluated our method on the RobotCar Seasons dataset, which contains over 20,000 daytime reference images and thousands of queries across various conditions, including night, rain, and snow.

Qualitative comparison: Our method successfully retrieves the correct daytime reference for nighttime queries, whereas the baseline DenseVLAD fails due to the domain gap.

Our approach sets a new benchmark for nighttime localization:

  • 300% Improvement: Compared to widely used VLAD-based methods, our condition-aware architecture achieved a nearly 3x jump in localization performance at night.
  • Outperforming Feature-Based Methods: On the "Night-All" benchmark, our method outperformed "Active Search" (a standard feature-based approach) by a factor of 1200% at coarse precision levels (5m, 10°).
  • Robustness: The method showed significant improvements for distance error thresholds above 3.5m, proving its utility for coarse-to-medium grain localization in challenging environments where other methods fail entirely.

Conclusion and Future Work

By explicitly modeling environmental conditions through specialized sub-networks, we have demonstrated that global retrieval can be made significantly more robust to extreme lighting changes. While retrieval-based methods currently provide coarse-to-medium precision, they offer a level of reliability at night that local feature matching cannot match. Future work will focus on "fine-grained" alignment, using these robust retrievals as a starting point for precise pose refinement, potentially combining the best of both retrieval and feature-based worlds.

Summary updated

Abstract

Outdoor visual localization is a crucial component to many computer vision systems. We propose an approach to localization from images that is designed to explicitly handle the strong variations in appearance happening between daytime and nighttime. As revealed by recent long-term localization benchmarks, both traditional feature-based and retrieval-based approaches still struggle to handle such changes. Our novel localization method combines a state-of-the-art image retrieval architecture with condition-specific sub-networks allowing the computation of global image descriptors that are explicitly dependent of the capturing conditions. We show that our approach improves localization by a factor of almost 300\% compared to the popular VLAD-based methods on nighttime localization.
BibTeX
@misc{germain2018improving,
  author = {Hugo Germain and Guillaume Bourmaud and Vincent Lepetit},
  title = {{Improving Nighttime Retrieval-Based Localization}},
  year = {2018},
  eprint = {1812.03707},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.03707},
}
Figure