Abstract
It has been shown that learning radiance fields with depth rendering and depth supervision can effectively promote the quality and convergence of view synthesis. However, this paradigm requires input RGB-D sequences to be synchronized, hindering its usage in the UAV city modeling scenario. As there exists asynchrony between RGB images and depth images due to high-speed flight, we propose a novel time-pose function, which is an implicit network that maps timestamps to $\rm SE(3)$ elements. To simplify the training process, we also design a joint optimization scheme to jointly learn the large-scale depth-regularized radiance fields and the time-pose function. Our algorithm consists of three steps: (1) time-pose function fitting, (2) radiance field bootstrapping, (3) joint pose error compensation and radiance field refinement. In addition, we propose a large synthetic dataset with diverse controlled mismatches and ground truth to evaluate this new problem setting systematically. Through extensive experiments, we demonstrate that our method outperforms baselines without regularization. We also show qualitatively improved results on a real-world asynchronous RGB-D sequence captured by drone.
The Challenge: Asynchronous Sensors in High-Speed Flight
Neural Radiance Fields (NeRFs) have revolutionized 3D reconstruction, but their reliance on synchronized multi-modal data is a major bottleneck for real-world deployment. In Unmanned Aerial Vehicle (UAV) scenarios, high flight speeds and varying sensor frequencies lead to temporal offsets. When an RGB image and a depth map are captured at different times but treated as a pair, the resulting "staggered" data leads to "ghosting" artifacts and incorrect geometry.
Methodology: The Time-Pose Function
The core of our solution is modeling the camera trajectory as a continuous, differentiable function rather than a discrete set of poses.
- Implicit Trajectory Modeling: We use a 1-D multi-resolution hash grid and a small MLP to map timestamps to 6-DoF poses. This allows the system to query the exact pose for any arbitrary timestamp, effectively "re-sampling" the sensors in software.
- Three-Step Optimization:
- Fitting: The time-pose function is initialized using poses from the RGB sequence (via SfM).
- Bootstrapping: A large-scale radiance field is initialized using RGB images to establish baseline geometry.
- Joint Refinement: The radiance field and time-pose function are jointly optimized. The differentiable nature of the trajectory allows depth maps to "pull" the estimated poses toward the correct geometric surfaces.
Experimental Results
We introduced the Asynchronous Urban Scene (AUS) Dataset, a synthetic benchmark with controlled temporal mismatches, to evaluate our framework.
- Superior View Synthesis: Our method outperformed Mega-NeRF by 1.11 dB in PSNR and significantly improved SSIM and LPIPS scores.
- Geometric Precision: Depth estimation error (RMSE) was reduced from 46.52 to 11.26 in challenging scenarios.
- Pose Correction: The joint optimization stage cut average pose error nearly in half (from 1.04m to 0.53m).
Conclusion
By treating the camera path as a continuous prior, we enable robust 3D reconstruction from asynchronous RGB-D streams. This framework is a significant step toward reliable large-scale city modeling using high-speed UAVs.
Summary updated