ExMaps: Long-Term Localization in Dynamic Scenes using Exponential Decay

Alexandros Rotsidis, Christof Lutteroth, Peter Hall, Christian RichardtWACV 2021 · DOI · OpenReview

TL;DR

This paper introduces "ExMaps," a novel method for robust long-term visual localization in dynamic environments like retail stores. By modeling the decay of feature point reliability over time using an exponential decay function, ExMaps maintains an up-to-date "live map" that significantly improves localization accuracy and speed compared to static maps.

What Problem Does This Paper Address?

Accurate camera localization is crucial for many applications, from robotics to augmented reality. Traditional methods often assume static environments, reconstructing a map once and using it indefinitely. However, real-world scenes are rarely static; they change due to lighting, weather, seasons, and, importantly for this paper, human activity and restocking in retail environments. These dynamic changes can render static maps unreliable, leading to localization failures. The challenge is to maintain accurate localization over extended periods in environments that are constantly evolving.

Methods

The core of ExMaps lies in its "live map" maintenance strategy. Instead of relying on a fixed, static map, ExMaps continuously updates a map by incorporating new observations. The key innovation is how it handles the dynamic nature of scenes:

  • Stability Score: Each 3D feature point in the map is assigned a "stability score." This score is based on its visibility over time, but crucially, it decays exponentially. This means points that haven't been observed recently become less reliable.
  • Exponential Decay: The decay is modeled using $N(t) = 2^{-\lambda t}$, inspired by radioactive decay. $\lambda$ is a decay constant, and $t$ is the time elapsed since the point was last observed. This allows the system to dynamically re-evaluate the usefulness of points.
  • Per-Session and Per-Image Scoring: The paper proposes two ways to apply this decay:
  • Per-Session Stability Score ($\sigma^s_p$): Weights all images within a session equally, with decay based on the session number.
  • Per-Image Stability Score ($\sigma^i_p$): Provides finer granularity by weighting more recent observations within a session more heavily.
  • Integration into Pose Estimation: These stability scores are then used to inform standard pose estimation algorithms like RANSAC and PROSAC. By prioritizing matches to more stable points, the system can filter out unreliable features caused by transient changes, leading to more robust localization.

Datasets

The authors evaluated their method on two datasets:

  • CMU Extended Seasons Dataset: An existing outdoor dataset with images captured over 12 months, allowing for evaluation in varying conditions (weather, seasons, lighting).
  • Retail Shop Dataset: A newly created indoor dataset specifically designed to capture the highly dynamic nature of retail environments. This dataset was collected over five days, with multiple visits per day to account for customer interaction and restocking.

Key Insights and Ideas

  • Time is a Feature: The paper effectively treats "time" as a crucial feature for localization, recognizing that a point's reliability isn't static but degrades over time in dynamic scenes.
  • Dynamic Environments Require Dynamic Maps: Static maps are a bottleneck for long-term localization. A "live map" that adapts to changes is essential.
  • Exponential Decay as a Proxy for Reliability: The choice of exponential decay is a clever way to model the decreasing likelihood of a feature point remaining consistent in a dynamic environment.
  • Retail Stores as a Challenging Benchmark: The paper highlights the unique challenges of retail environments (frequent restocking, shopper interaction) and introduces a dataset to address this gap in existing benchmarks.

Key Results

  • Improved Localization Accuracy: The live map approach consistently reduced translation and rotation errors compared to using a static base map across both datasets. For example, on the retail shop dataset, translation errors were significantly lower with the live map.
  • Increased Robustness (More Inliers): The live map approach consistently yielded more inlier matches, indicating more reliable correspondences were found. This also allowed some query images that failed to localize in the base map to be successfully localized in the live map.
  • Faster Performance: While not the primary goal, the live map approach, especially with PROSAC, showed a reduction in run time as more sessions were integrated.
  • PROSAC with Stability Scores Outperforms Baselines: The proposed PROSAC version using the $\sigma^i$-LR score (combining per-image stability and Lowe's ratio) consistently returned more inliers and lower pose errors than other PROSAC versions and vanilla RANSAC.

Limitations and Future Work

  • Frequent Data Input: The continuous improvement of the live map relies on frequent data input from users, which might be a limitation in some scenarios.
  • Uniform Session Localization: The live map's effectiveness depends on new session data being able to uniformly localize. If only segments of a session localize, the live map might contain outdated sections.
  • Bootstrap Phase: Collecting data over a full year to capture seasonal changes could serve as a valuable "bootstrap" phase for initial map creation.
  • Future Work: The authors suggest exploring adding new points or compressing existing maps based on stability scores. They also plan to release their source code and the retail shop dataset.

Summary updated

Abstract

Visual camera localization using offline maps is widespread in robotics and mobile applications. Most state-of-the-art localization approaches assume static scenes, so maps are often reconstructed once and then kept constant. However, many scenes are dynamic and as changes in the scene happen, future localization attempts may struggle or fail entirely. Therefore, it is important for successful long-term localization to update and maintain maps as new observations of the scene, and changes in it, arrive. We propose a novel method for automatically discovering which points in a map remain stable over time, and which are due to transient changes. To this end, we calculate a stability store for each point based on its visibility over time, weighted by an exponential decay over time. This allows us to consider the impact of time when scoring points, and to distinguish which points are useful for long-term localization. We evaluate our method on the CMU Extended Seasons dataset (outdoors) and a new indoor dataset of a retail shop, and show the benefit of maintaining a `live map' that integrates updates over time using our exponential decay based method over a static `base map'.
BibTeX
@inproceedings{rotsidis2021exmaps,
  author = {Alexandros Rotsidis and Christof Lutteroth and Peter Hall and Christian Richardt},
  title = {{ExMaps: Long-Term Localization in Dynamic Scenes using Exponential Decay}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00291},
}
Figure