BEVRender: Vision-based Cross-view Vehicle Registration in Off-road GNSS-denied Environment

Lihong Jin, Wei Dong, Wenshan Wang, Michael KaessIROS 2024 · DOI

TL;DR

This paper introduces BEVRender, a novel vision-based system for accurately localizing ground vehicles in off-road environments where GPS signals are unreliable. By generating high-quality local Bird's-Eye-View (BEV) images and matching them against aerial maps, BEVRender achieves robust and precise localization without relying on extensive pre-stored datasets.

What Problem Does This Paper Address?

Autonomous vehicles rely heavily on accurate localization to navigate safely. While Global Navigation Satellite Systems (GNSS) like GPS provide global positioning, their signals are often blocked in challenging environments such as dense forests, urban canyons, or tunnels, leading to temporary or persistent localization failures. Vision-based localization (VBL) offers an alternative, but off-road scenarios present unique difficulties: a lack of distinct visual landmarks (like roads or buildings) and unstable vehicle poses due to uneven terrain. Existing VBL methods often struggle with these challenges. Some methods require large storage for image retrieval, while others are susceptible to drift from visual-inertial odometry (VIO) systems. This paper aims to bridge this gap by providing a robust and efficient VBL solution specifically for GNSS-denied off-road environments.

Methods

BEVRender employs a learning-based approach that combines a novel BEV image generation module with classical template matching. The core of the system involves:

  • BEV Image Generation: The system leverages a modified BEVFormer architecture for feature encoding. This involves a Vision Transformer (ViT) backbone to process camera inputs. A key innovation is the integration of a deformable attention module, which allows for more adaptive and precise feature extraction by learning offsets to reference points. This module captures temporal information across consecutive frames, creating a rich BEV representation.
  • Rendering Head: A dedicated rendering head, implemented as a convolutional neural network (CNN), translates the encoded BEV features into high-quality, top-down BEV images of the surrounding terrain. This ensures that critical spatial information is preserved for accurate localization.
  • Template Matching: The generated BEV image is then used for localization by performing template matching against a georeferenced aerial map. This process aligns the vehicle's local view with the global map to determine its precise position. This approach avoids the need for large on-board image retrieval databases.

Datasets

The authors collected their own real-world dataset in the Pittsburgh area using a vehicle equipped with trinocular cameras and an IMU. This dataset consists of several sequences with ground truth GPS information (x, y, and azimuth angle). For training, they use trinocular RGB images, timestamp, and GPS ground truth. For testing, they crop and rotate the aerial map based on GPS ground truth to create a search region. The paper does not introduce a new benchmark dataset but uses its collected data for evaluation.

Key Insights and Ideas

  • BEV as a Bridge: Generating high-quality local BEV images acts as a crucial intermediary, translating complex 3D visual information into a format directly comparable to 2D aerial maps.
  • Deformable Attention for Robustness: The use of deformable attention in the encoder is key to handling the visual instability and varying terrain in off-road environments. It allows the model to adapt its feature sampling to local conditions.
  • Efficiency over Storage: By generating BEV images on-the-fly and using template matching, BEVRender significantly reduces the storage requirements compared to traditional image retrieval methods, making it more practical for on-board deployment.
  • VIO Drift Mitigation: While acknowledging the inherent drift in VIO systems, the approach aims to correct these drifts by leveraging the global context provided by the aerial map, rather than solely relying on cumulative VIO estimates.

Key Results

  • Improved Accuracy: BEVRender demonstrates superior localization accuracy compared to existing GNSS-denied visual localization methods like Litman [3] and GeoDTR [7]. For instance, on Seq 1, their method achieved a mean error of 19.33 meters with a standard deviation of 26.09 meters, outperforming Litman [3] (24.35m mean, 13.50m std) and GeoDTR [7] (top 1: 82.72m mean, 25.52m std; top 5 avg: 67.35m mean, 24.22m std).
  • High Registration Frequency: The system achieves a high matching frequency, reporting a match rate of 57.50% for Seq 4, indicating its ability to provide consistent localization updates.
  • Generalization: The method shows good generalization capabilities, performing well on unseen trajectories and environments.
  • Efficiency: The system is computationally efficient, with a localization time of approximately 0.12 seconds per camera frame, supporting real-time operation at 3 frames per second.

Limitations and Future Work

  • Rendering Quality: The authors acknowledge that in some failure cases, the rendered BEV images are of moderate visual quality, which can lead to NCC failing to find a match.
  • Seasonal Variations: The current system's ability to generalize across different seasons is limited. Future work will focus on improving the network's capacity to learn and generalize features from varied seasonal landscapes.
  • Diffusion Models for Detail: To enhance the detail and precision of BEV images, the authors plan to explore incorporating diffusion modules, inspired by diffusion transformers.
  • GPS Independence: Further research aims to reduce dependence on GPS for training by leveraging local state estimates from VIO.
  • Learnable Template Matching: The authors anticipate exploring learnable template matching to overcome the limitations of NCC's uniform pixel weighting, which can lead to failures in certain scenarios.

Summary updated

BibTeX
@inproceedings{jin2024bevrender,
  author = {Lihong Jin and Wei Dong and Wenshan Wang and Michael Kaess},
  title = {{BEVRender: Vision-based Cross-view Vehicle Registration in Off-road GNSS-denied Environment}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10802702},
}
Figure