Abstract
High-fidelity 3D reconstruction of road scenes is essential for the development and testing of autonomous driving systems. While 3D Gaussian Splatting (3D-GS) has emerged as a leading technique for real-time rendering, ground-level data collection often suffers from limited fields of view and significant occlusions. This paper introduces UC-GS (Uncertainty-aware Cross-view Gaussian Splatting), a novel framework that integrates aerial drone imagery to supplement ground-level views. By projecting rendering uncertainty from ground-based cameras onto aerial perspectives, UC-GS selectively utilizes drone data to fill in missing information without introducing noise from view disparity. Our experiments on high-quality synthetic benchmarks demonstrate that UC-GS significantly improves reconstruction quality, particularly in challenging out-of-distribution camera perspectives.
Introduction
In the quest for realistic autonomous driving simulations, 3D Gaussian Splatting (3D-GS) has become a preferred method due to its efficiency and high-quality output. However, 3D-GS is highly dependent on the coverage of the input data. Ground vehicles, the standard source for mapping, are restricted by their physical height and perspective, leading to "blind spots" in complex urban environments.
While aerial imagery from drones offers a potential solution to these coverage gaps, naively combining ground and aerial data is problematic. The "view disparity" between a car bumper and a drone flying at 20 meters is vast; aerial views capture rooftops and global context but lack the fine-grained texture of the road surface, while ground views capture details that appear as noise from above. This mismatch often leads to poor model convergence and blurred textures in the final reconstruction.
Methodology: The UC-GS Framework
To address these challenges, we propose UC-GS, which treats ground-level data as the primary "teacher" and aerial data as a specialized "assistant." The framework operates through three core stages:
1. Ensemble-based Uncertainty Estimation
We first train an ensemble of multiple 3D-GS models using only ground-level images. By calculating the variance in pixel color across these models, we generate a "ground uncertainty map." High variance indicates regions where the ground-level data is insufficient to produce a consistent reconstruction, effectively identifying the "blind spots" of the vehicle.
2. Cross-view Projection
To guide the drone's contribution, we project this ground-level uncertainty into the aerial perspective. We utilize DUSt3R, a state-of-the-art dense stereo model, to establish pixel-to-pixel correspondences between the car's viewpoint and the drone's viewpoint. This allows the system to map exactly which pixels in an aerial image correspond to high-uncertainty regions on the ground.
3. Uncertainty-aware Training
Finally, we incorporate this projected uncertainty into the loss function during the joint training of the final model. Aerial pixels that correspond to high-uncertainty ground regions are assigned higher weights, forcing the model to learn from the drone in those specific areas. Conversely, irrelevant aerial data—such as the tops of buildings or distant vistas not visible from the road—is down-weighted to prevent it from degrading the road-level reconstruction.
Experimental Setup
Due to the difficulty of obtaining perfectly synchronized real-world drone and car datasets, we evaluated UC-GS using a high-fidelity synthetic benchmark created in AirSim and Unreal Engine.
- Environments: Detailed digital twins of New York City (NYC) and San Francisco (SF).
- Data Composition: Each scene utilized 315 ground-level images (captured at 1.5m and 1.8m) and 351 aerial images (captured at 20m).
- Evaluation Metrics: We measured performance using Peak Signal-to-Noise Ratio (PSNR) on held-out views and "shifted" views (simulating camera movements common in autonomous driving).
Results and Discussion
UC-GS consistently outperformed state-of-the-art baselines, including Scaffold-GS, Mip-Splatting, and Nerfacto.
Quantitative Performance
In the NYC dataset, UC-GS achieved a PSNR improvement of 0.68 dB on standard held-out tests. More importantly, in scenarios involving shifted or rotated views—where the camera moves outside the original training trajectory—UC-GS improved PSNR by nearly 0.90 dB compared to models trained on ground data alone. This indicates that the aerial data successfully "filled in" the geometry that ground-only models could not predict.
Qualitative Improvements
The visual results show a marked difference in clarity. Road textures, lane markings, and roadside signage—which typically appear "melted" or distorted in standard 3D-GS due to lack of multi-view consistency—are rendered with sharp edges and accurate textures in UC-GS. The ablation studies further revealed that normalizing the uncertainty (specifically using an $n$-th root where $n \ge 6$) was vital for balancing the influence of the two views.
Conclusion and Future Work
UC-GS demonstrates that aerial imagery can significantly enhance road-level 3D reconstructions when integrated through an uncertainty-aware framework. By using drones as a targeted "assistant" to fill ground-level gaps, we achieve higher fidelity and better generalization for autonomous driving simulations.
Future research will focus on transitioning this method to real-world datasets, where challenges such as dynamic objects (pedestrians and traffic) and varying lighting conditions between drone and car captures must be addressed. Additionally, we aim to optimize the projection process to reduce the computational overhead of dense pixel matching.
Summary updated