Abstract
Large-scale radiance fields are promising mapping tools for smart transportation applications like autonomous driving or drone delivery. However, for large-scale scenes, compact synchronized RGB-D cameras are often inapplicable due to limited sensing range, and using separate RGB and depth sensors inevitably leads to unsynchronized sequences. We propose AsyncNeRF, the first solution that self-calibrates the mismatch between RGB and depth frames. By modeling the camera trajectory as a continuous time-pose function, we enable joint optimization of the scene representation and sensor poses. Benchmarking on a diverse, photorealistic dataset demonstrates that AsyncNeRF significantly outperforms prior counterparts in both reconstruction quality and geometric accuracy.
The Challenge of Asynchronous RGB-D Mapping
Incorporating depth data into Neural Radiance Fields (NeRFs) is a proven way to speed up convergence and eliminate "floaters" (artifacts in empty space). However, existing methods assume that RGB images and depth maps are perfectly synchronized. In real-world scenarios, especially with Unmanned Aerial Vehicles (UAVs) flying at high speeds, hardware or software delays often lead to temporal asynchrony. If a depth map is captured even a few milliseconds after its corresponding RGB image, the camera has already moved. Using these misaligned pairs for supervision leads to blurry reconstructions and geometric errors.
Methodology: The Time-Pose Function
The core innovation of AsyncNeRF is the Time-Pose Function ($\phi$), an implicit neural network that maps a single timestamp ($t$) to a 6-DoF camera pose ($SE(3)$). This approach leverages the domain-specific fact that while sensors may be asynchronous, they are sampled from the same continuous physical trajectory.
The optimization pipeline consists of three stages:
- Time-Pose Function Fitting: The model learns the drone's trajectory by fitting the function to initial poses obtained from Structure-from-Motion (SfM) on the RGB sequence.
- Radiance Field Bootstrapping: A large-scale NeRF is initialized using RGB images and the poses predicted by the time-pose function.
- Joint Optimization: The system jointly optimizes the scene representation and the time-pose function. Because the pipeline is fully differentiable, depth maps provide supervision at their exact capture timestamps, "nudging" the trajectory to align perfectly with observed data.
Experimental Validation
To validate the approach, the authors introduced the Asynchronous Urban Scene (AUS) Dataset, a large-scale synthetic benchmark built in AirSim. It includes six diverse environments (e.g., New York, San Francisco) and various trajectory types. Asynchrony was simulated using both fixed offsets and random temporal jitter. Real-world validation was also performed using a DJI M300 UAV equipped with high-definition cameras and LiDAR.
Key Results and Performance
AsyncNeRF significantly outperformed baselines like Mega-NeRF and NeRF-W on the AUS dataset across all metrics:
- Image Quality: Achieved a +1.11 dB improvement in PSNR over Mega-NeRF.
- Geometry: The Root Mean Square Error (RMSE) for depth dropped from 46.52 to 11.26, a ~75% reduction in error.
- Pose Accuracy: Joint optimization cut depth sensor pose error in half, reducing translation error from 1.07m to 0.53m.
Qualitatively, the method produces much sharper depth maps and avoids the "smearing" artifacts typically seen in glass or thin structures when using misaligned data. By treating the trajectory as a continuous function rather than discrete points, AsyncNeRF provides a robust solution for high-speed robotic mapping.
Summary updated