DisPlacing Objects: Improving Dynamic Vehicle Detection via Visual Place Recognition under Adverse Conditions

Stephen Hausler, Sourav Garg, Punarjay Chakravarty, Shubham Shrivastava, Ankit Vora, Michael Milford

Abstract

Standard 2D object detection models often struggle in adverse environmental conditions such as heavy rain, snow, or low-light environments. This paper presents a novel framework that leverages Visual Place Recognition (VPR) to enhance vehicle detection. By comparing current query images with previously recorded reference maps, the system identifies discrepancies to filter out false positives caused by environmental noise. Experimental results on the Oxford RobotCar and Dark Zurich datasets demonstrate significant improvements in precision and F1 scores, particularly in challenging nighttime and wet-weather scenarios.

1. Introduction

Autonomous vehicles (AVs) primarily rely on vision-based object detectors like YOLO to navigate urban environments. While these models perform exceptionally well under clear conditions, their reliability degrades significantly in "adverse conditions"—weather or lighting states that deviate from typical training data. In these scenarios, detectors often suffer from low recall (missing actual vehicles) or low precision (identifying shadows, reflections, or glare as "ghost" vehicles).

This research explores the hypothesis that localization can directly aid perception. By utilizing the spatial context provided by a prior map, an AV can effectively "look up" what a scene should look like in its static state. By identifying differences between the current view and a clean reference image, the system can more accurately distinguish between permanent infrastructure and dynamic objects.

2. The DisPlacing Objects Framework

The authors propose a two-stage pipeline designed to operate without the need for high-fidelity 3D maps or perfect pixel-level alignment.

2.1 Candidate Generation

The system first employs an off-the-shelf detector, YOLOP, configured with a very low confidence threshold. This strategy prioritizes high recall, ensuring that nearly every potential vehicle is captured, albeit at the cost of introducing significant noise and false positives.

2.2 VPR Retrieval and Feature Extraction

To provide context, the system uses Patch-NetVLAD to retrieve a matching reference image from a pre-existing database (the map). This reference image represents the same geographic location under clear conditions. To maintain computational efficiency, the system reuses the CNN backbone features generated during the VPR step for the subsequent classification task.

2.3 Map-Matching Classifier

For every candidate detection box in the query image, the system examines the corresponding coordinates in the reference map. A lightweight Multi-Layer Perceptron (MLP) compares the features from both regions. The logic is straightforward: if the regions are significantly different, the detection is likely a true dynamic vehicle. If the regions are similar, the detection is flagged as a false positive—likely a static environmental feature like a mailbox or a specific shadow pattern.

2.4 Score Fusion

The final confidence score for a detection is calculated by averaging the original YOLOP score with the new map-matching score, providing a robust filter against environmental artifacts.

3. Experimental Setup

The framework was evaluated using two distinct benchmarks:

  • Oxford RobotCar (RCSv2): The primary training and evaluation set. The authors manually annotated 1,122 image pairs with centroid flags for cars, trucks, and buses to facilitate "weakly supervised" learning.
  • Dark Zurich: A dataset used to test the model's generalization capabilities. The model trained on Oxford data was tested on Zurich's nighttime streets without further fine-tuning.

4. Key Results and Insights

4.1 Performance Gains

The proposed method consistently outperformed the baseline YOLOP detector. At a 95% recall rate, the average precision across all test traverses increased from 73.3% to 81.1%. The system proved especially adept at removing false positives caused by glare and reflections on wet roads at night.

4.2 The "Same Map" Advantage

The system's performance peaks when the AV traverses a route it has previously mapped. In "Night, Rear, Same map" tests, the F1 score improved from 60.4 to 68.6, suggesting that local memory is a highly effective tool for cleaning sensor noise.

4.3 Generalization and VPR Robustness

On the Dark Zurich dataset, the model improved the F1 score from 71.8 to 74.3. This indicates that the system is not merely memorizing specific locations but is learning a generalizable logic for comparing query images against reference maps. Furthermore, the study found that coarse VPR retrieval is sufficient; the system does not require precise 6-DoF metric localization to be effective.

5. Limitations and Future Work

While promising, the system has specific dependencies:

  • Localization Accuracy: If the VPR engine retrieves the wrong street, the map-matching logic fails.
  • Static Map Assumption: The model assumes the reference map is static. Parked cars present during the mapping phase but absent during the query phase can cause confusion.
  • Object Diversity: Currently limited to vehicles, the system occasionally misidentifies groups of pedestrians as vehicles because they represent a "difference" from the map. Future iterations could expand the classifier to include pedestrians, cyclists, and other dynamic classes.

Summary updated

BibTeX
@inproceedings{hausler2023displacing,
  author = {Stephen Hausler and Sourav Garg and Punarjay Chakravarty and Shubham Shrivastava and Ankit Vora and Michael Milford},
  title = {{DisPlacing Objects: Improving Dynamic Vehicle Detection via Visual Place Recognition under Adverse Conditions}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  doi = {10.1109/iros55552.2023.10341550},
}
Figure