Practical Visual Localization for Autonomous Driving: Why Not Filter?

Anh-Dzung Doan, Thanh-Toan Do, Yasir Latif, Tat-Jun Chin, Ian D. Reid

Abstract

A major focus of current research on place recognition is visual localization for autonomous driving. In this scenario, as cameras will be operating continuously, it is realistic to expect videos as an input to visual localization algorithms, as opposed to the single-image querying approach used in other visual localization works. In this paper, we show that exploiting temporal continuity in the testing sequence significantly improves visual localization - qualitatively and quantitatively. Although intuitive, this idea has not been fully explored in recent works. To this end, we propose two filtering approaches to exploit the temporal smoothness of image sequences: i) filtering on discrete domain with Hidden Markov Model, and ii) filtering on continuous domain with Monte Carlo-based visual localization. Our approaches rely on local features with an encoding technique to represent an image as a single vector. The experimental results on synthetic and real datasets show that our proposed methods achieve better results than state of the art (i.e., deep learning-based pose regression approaches) for the task on visual localization under significant appearance change.

Introduction

For an autonomous vehicle to navigate safely, it must determine its exact 6 Degree of Freedom (6-DoF) pose relative to a map. Visual localization matches current camera feeds against a pre-recorded visual map. However, appearance change—caused by weather, seasons, or lighting—remains a significant hurdle. A street mapped on a sunny afternoon looks vastly different during a snowy night.

While many deep learning methods (e.g., PoseNet, MapNet) attempt to learn these variations from single images, they often overfit or fail in extreme conditions. This paper argues that since vehicles operate continuously, localization systems should leverage video sequences. By exploiting the temporal smoothness of a car's motion, we can filter out ambiguities caused by transient appearance changes.

Methodology

The proposed system uses a robust image encoder as the foundation for two probabilistic filtering frameworks.

The Observation Encoder

To handle appearance shifts, raw images are not fed directly into a network. Instead, dense SIFT features are extracted and embedded using VLAD (Vector of Locally Aggregated Descriptors). The resulting vectors undergo PCA projection and whitening. This critical step suppresses "background" features (sky, road surface) that are ubiquitous but uninformative for localization, allowing the system to focus on stable structural landmarks.

Filtering on a Discrete Domain (HMM)

The first approach employs a Hidden Markov Model (HMM). The pre-recorded map is treated as a discrete set of "places" (image indices). As the query video progresses, the HMM updates the probability of the vehicle being at any specific place. The transition matrix is constrained by the vehicle's maximum physical speed, preventing impossible spatial jumps.

The HMM-based visual localization pipeline, showing the transition from image sequences to belief distribution over map locations.

Filtering on a Continuous Domain (MCVL)

The second approach uses Monte Carlo Visual Localization (MCVL), a particle filter. Given that cars are largely confined to 2D road networks, a motion model tracks linear and angular velocity. Particles representing potential 6-DoF poses are propagated by the motion model and weighted based on how well their predicted view matches the camera feed via the observation encoder.

The Monte Carlo Visual Localization (MCVL) pipeline, detailing the particle filtering process and motion model integration.

Experimental Setup

The methods were evaluated on two primary datasets:

  • Synthetic Dataset: Built using the Grand Theft Auto V (GTA V) engine, featuring 59 vehicles across 3.36 km² under varied conditions (snow, fog, rain, night).
  • Oxford RobotCar: A real-world benchmark. Training and testing sequences were recorded months apart to ensure massive appearance differences.

Results and Discussion

The experimental results highlight several key insights:

  • Temporal Continuity is Essential: Tracking sequences resolves ambiguities that baffle single-frame methods. Filtering handles temporary "blind spots" (e.g., heavy glare) naturally.
  • Classical Filtering vs. Deep Learning: Traditional probabilistic filters combined with robust embeddings outperformed end-to-end CNN pose regressors like MapNet and PoseNet. Neural networks struggled with severe appearance shifts and repetitive structures where filtering maintained tracking.
  • Trajectory Characteristics: While HMM slightly outperformed MCVL in raw frame-level accuracy in some tests, MCVL produced smoother, more realistic continuous trajectories, whereas HMM tended to "snap" between discrete database frames.
  • Efficiency: The sum-pooling + PCA/whitening encoder achieved high accuracy with low computational overhead, making it suitable for real-time inference.

Conclusion

This work demonstrates that exploiting temporal continuity through classical filtering significantly improves visual localization under extreme appearance changes. Future work will explore replacing handcrafted SIFT features with deep learning-based local extractors and developing scalable indexing for country-scale maps. Additionally, quantifying uncertainty remains a priority for safety-critical autonomous driving applications.

Summary updated

Abstract

A major focus of current research on place recognition is visual localization for autonomous driving. In this scenario, as cameras will be operating continuously, it is realistic to expect videos as an input to visual localization algorithms, as opposed to the single-image querying approach used in other visual localization works. In this paper, we show that exploiting temporal continuity in the testing sequence significantly improves visual localization - qualitatively and quantitatively. Although intuitive, this idea has not been fully explored in recent works. To this end, we propose two filtering approaches to exploit the temporal smoothness of image sequences: i) filtering on discrete domain with Hidden Markov Model, and ii) filtering on continuous domain with Monte Carlo-based visual localization. Our approaches rely on local features with an encoding technique to represent an image as a single vector. The experimental results on synthetic and real datasets show that our proposed methods achieve better results than state of the art (i.e., deep learning-based pose regression approaches) for the task on visual localization under significant appearance change. Our synthetic dataset and source code are made publicly available at https://sites.google.com/view/g2d-software/home and https://github.com/dadung/Visual-Localization-Filtering.
BibTeX
@article{doan2020practical,
  author = {Anh-Dzung Doan and Thanh-Toan Do and Yasir Latif and Tat-Jun Chin and Ian D. Reid},
  title = {{Practical Visual Localization for Autonomous Driving: Why Not Filter?}},
  journal = {Neural Computing and Applications},
  year = {2020},
  doi = {10.1007/s00521-020-05339-y},
}
Figure