SeqNetVLAD vs PointNetVLAD: Image Sequence vs 3D Point Clouds for Day-Night Place Recognition

Sourav Garg, Michael Milford

Introduction

Visual Place Recognition (VPR) is a fundamental capability for autonomous robots, enabling them to localize by recognizing previously visited locations. However, VPR faces significant challenges when environments undergo drastic appearance changes, such as the transition from day to night. Traditionally, researchers have relied on either 3D point clouds (LiDAR), which offer geometric stability but sparse data, or single RGB images, which provide rich information but are highly sensitive to lighting and shadows.

This paper investigates whether temporal sequences of images, which implicitly capture spatial structure through movement, can match or exceed the performance of explicit 3D point cloud data in these challenging scenarios.

Methods

The study compares two primary architectures: PointNetVLAD and SeqNetVLAD. PointNetVLAD is a deep learning framework designed to extract global descriptors from 3D point clouds. In contrast, SeqNetVLAD leverages "sequential descriptors" derived from short bursts of images rather than single frames.

To ensure a fair comparison, the authors introduced the "Metric Span" concept. While PointNetVLAD typically uses a 20-meter submap, SeqNetVLAD was configured to use a sequence of 5 images spaced 2 meters apart, covering a 10-meter span—effectively using less spatial information than the 3D baseline.

Overview of the SeqNetVLAD pipeline showing sequence-based descriptors and sequential score aggregation for day-night localization.

Additionally, the authors evaluated SeqNetVLAD-HVPR, a hierarchical pipeline that uses sequential descriptors for initial candidate retrieval followed by refinement using single-image descriptors.

Datasets and Benchmarking

Evaluations were conducted using the Oxford Robotcar dataset, focusing on day and night traverses. A key contribution of this work is the refinement of the benchmarking process. The authors identified that standard train/test splits in previous 3D-based studies allowed for visual overlap between training and testing areas (where the camera could "see" into the test area from the training area). They proposed an updated split to ensure no visual overlap, providing a more rigorous test for image-based methods.

Results and Discussion

The experimental results demonstrate that sequence-based image descriptors significantly outperform both single-image methods and the 3D point cloud baseline. As shown in the results table, SeqNetVLAD (S5) achieved a Recall@1 of 0.87, compared to 0.77 for PointNetVLAD (Base).

Comparative results on the Oxford Robotcar dataset across single image, point cloud, and sequential descriptors.

These findings suggest that deep networks are highly capable of extracting implicit geometric cues—such as parallax and perspective changes—from image sequences. The high information density of RGB sensors allows these sequences to overcome the disadvantages of changing illumination. Furthermore, the temporal consistency provided by sequences acts as a robust prior, filtering out noise caused by appearance shifts.

Conclusion

This research challenges the assumption that explicit 3D data is inherently superior for robot localization in changing environments. By leveraging the temporal dimension, image-based sequences can provide superior localization performance. Future work will explore the fusion of 2D and 3D modalities to create even more robust spatial understanding systems across diverse and repetitive environments.

Summary updated

Abstract

Place Recognition is a crucial capability for mobile robot localization and navigation. Image-based or Visual Place Recognition (VPR) is a challenging problem as scene appearance and camera viewpoint can change significantly when places are revisited. Recent VPR methods based on ``sequential representations'' have shown promising results as compared to traditional sequence score aggregation or single image based techniques. In parallel to these endeavors, 3D point clouds based place recognition is also being explored following the advances in deep learning based point cloud processing. However, a key question remains: is an explicit 3D structure based place representation always superior to an implicit ``spatial'' representation based on sequence of RGB images which can inherently learn scene structure. In this extended abstract, we attempt to compare these two types of methods by considering a similar ``metric span'' to represent places. We compare a 3D point cloud based method (PointNetVLAD) with image sequence based methods (SeqNet and others) and showcase that image sequence based techniques approach, and can even surpass, the performance achieved by point cloud based methods for a given metric span. These performance variations can be attributed to differences in data richness of input sensors as well as data accumulation strategies for a mobile robot. While a perfect apple-to-apple comparison may not be feasible for these two different modalities, the presented comparison takes a step in the direction of answering deeper questions regarding spatial representations, relevant to several applications like Autonomous Driving and Augmented/Virtual Reality. Source code available publicly https://github.com/oravus/seqNet.
BibTeX
@misc{garg2021seqnetvlad,
  author = {Sourav Garg and Michael Milford},
  title = {{SeqNetVLAD vs PointNetVLAD: Image Sequence vs 3D Point Clouds for Day-Night Place Recognition}},
  year = {2021},
  eprint = {2106.11481},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.11481},
}
Figure