Video2BEV: Transforming Drone Videos to BEVs for Video-based Geo-localization

Hao Ju, Zhedong Zheng

Introduction

Visual geo-localization for drones—the process of identifying a drone's geographic coordinates by matching its camera feed against a database of satellite imagery—is a critical capability for autonomous navigation in GPS-denied environments. Traditionally, this has been approached as an image-to-image retrieval task. However, single-image matching is plagued by two fundamental issues: severe occlusions from urban structures and a significant viewpoint gap between oblique drone cameras and nadir (top-down) satellite sensors.

Video2BEV introduces a paradigm shift by moving from single-frame matching to leveraging the temporal and geometric richness of video sequences. By reconstructing the scene into a Bird’s Eye View (BEV) using 3D Gaussian Splatting (3DGS), the framework effectively "sees around" obstacles and aligns the drone's perspective with satellite imagery, setting new performance standards in the field.

The Challenges of Image-Based Geo-localization

Current drone geo-localization methods often fail in dense urban environments. When a drone flies at low altitudes, buildings, trees, and other vertical structures frequently occlude the ground-level landmarks necessary for accurate matching. Furthermore, the perspective distortion between a drone's slanted view and a satellite's top-down view is non-trivial. While 2D geometric transformations like polar warping have been used to bridge this gap, they often produce blurry or distorted results that discard fine-grained spatial details.

Figure 1: Comparison between image-based and video-based query. While a single image might be occluded, a video sequence provides multiple viewpoints that allow for unoccluded scene reconstruction and successful matching.

The Video2BEV Framework

The Video2BEV approach addresses these challenges through three innovative components:

1. 3DGS-based Video2BEV Transformation

Instead of relying on 2D warping, Video2BEV utilizes 3D Gaussian Splatting (3DGS) to reconstruct a 3D representation of the scene from multi-view video frames. This 3D model is then projected into a clean, undistorted BEV. This process not only resolves the viewpoint disparity but also mitigates occlusion issues, as the moving drone captures different parts of the scene over time, allowing the 3DGS to "fill in" areas that are hidden in any single frame.

2. Hard Negative Sample Synthesis

To improve the model's discriminative power, the authors developed a synthesis module using Stable Diffusion and LoRA, guided by a Visual-LLM. This module generates "hard negatives"—synthetic BEV images that are visually similar to the target location but represent incorrect coordinates. Training with these challenging examples forces the model to focus on subtle, unique environmental features rather than broad color patterns or general shapes.

3. Two-Stage Optimization

The retrieval process is optimized in two stages. First, a shared Vision Transformer (ViT-S) encoder extracts global features to identify a set of top candidates via contrastive learning. Second, a dedicated matching module (MLP) performs fine-grained re-ranking of these candidates, ensuring that the final localization is as precise as possible.

Experimental Evaluation and Results

To evaluate the framework, the researchers introduced UniV, a large-scale video-based geo-localization dataset extending the University-1652 benchmark. UniV includes 3,304 drone videos across 1,652 locations, specifically incorporating a challenging 30° elevation angle to simulate realistic low-altitude urban flights.

The results demonstrate the clear superiority of the video-based approach. On the UniV dataset, Video2BEV achieved a state-of-the-art Average Precision (AP) of 96.80% for drone-to-satellite retrieval.

Figure 6: Performance comparison across different baseline models. Switching from image-to-image (blue) to video-to-image (red) consistently yields significant improvements in Average Precision (AP).

Key findings from the experiments include:

  • Temporal Advantage: Converting existing image-based models (such as LPN or FSRA) to use video-based BEV inputs resulted in an average AP increase of over 24%.
  • Low-Altitude Robustness: At the difficult 30° elevation angle, Video2BEV outperformed competing methods by significant margins (3.2% to 5.8% AP), proving its resilience to heavy occlusion.
  • Generalization: The model demonstrated strong zero-shot performance on the SUES-200 dataset, achieving 91.50% AP without fine-tuning, which indicates that the learned features are robust across different geographic regions.

Conclusion and Future Directions

Video2BEV demonstrates that the transition from image-based to video-based geo-localization is essential for robust drone navigation. By integrating 3D scene reconstruction into the retrieval pipeline, the framework effectively eliminates the viewpoint gap and overcomes the limitations of single-frame occlusions.

While highly effective, the current system faces challenges in adverse weather conditions, such as heavy rain, which can introduce noise into the 3DGS reconstruction. Future research will focus on enhancing the robustness of 3DGS to environmental noise and optimizing the computational efficiency of the BEV transformation for real-time deployment on drone hardware.

Summary updated

Abstract

Existing approaches to drone visual geo-localization predominantly adopt the image-based setting, where a single drone-view snapshot is matched with images from other platforms. Such task formulation, however, underutilizes the inherent video output of the drone and is sensitive to occlusions and viewpoint disparity. To address these limitations, we formulate a new video-based drone geo-localization task and propose the Video2BEV paradigm. This paradigm transforms the video into a Bird's Eye View (BEV), simplifying the subsequent \textbf{inter-platform} matching process. In particular, we employ Gaussian Splatting to reconstruct a 3D scene and obtain the BEV projection. Different from the existing transform methods, \eg, polar transform, our BEVs preserve more fine-grained details without significant distortion. To facilitate the discriminative \textbf{intra-platform} representation learning, our Video2BEV paradigm also incorporates a diffusion-based module for generating hard negative samples. To validate our approach, we introduce UniV, a new video-based geo-localization dataset that extends the image-based University-1652 dataset. UniV features flight paths at $30^\circ$ and $45^\circ$ elevation angles with increased frame rates of up to 10 frames per second (FPS). Extensive experiments on the UniV dataset show that our Video2BEV paradigm achieves competitive recall rates and outperforms conventional video-based methods. Compared to other competitive methods, our proposed approach exhibits robustness at lower elevations with more occlusions.
BibTeX
@inproceedings{ju2025video,
  author = {Hao Ju and Zhedong Zheng},
  title = {{Video2BEV: Transforming Drone Videos to BEVs for Video-based Geo-localization}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure