Introduction
Neural Radiance Fields (NeRFs) have revolutionized 3D scene reconstruction by enabling high-fidelity novel view synthesis from sparse image sets. Recent research has demonstrated that incorporating depth information (RGB-D) significantly enhances reconstruction quality and accelerates convergence by providing explicit geometric constraints. However, a significant practical challenge remains: sensor asynchrony. In real-world applications, particularly those involving Unmanned Aerial Vehicles (UAVs), RGB cameras and depth sensors (e.g., LiDAR) often operate at different frequencies due to hardware and power constraints.
As illustrated in Figure 3(a), this temporal mismatch means that the camera's pose at timestamp $t_{rgb}$ does not coincide with the sensor's pose at $t_{depth}$. Directly supervising a NeRF with misaligned depth maps leads to geometric blurring and artifacts. To address this, we introduce AsyncNeRF, a framework that models the drone's trajectory as a continuous function to self-align asynchronous data during training.
Methodology
The AsyncNeRF pipeline consists of three primary components designed to bridge the temporal gap between sensors.
1. Continuous Time-Pose Function
Instead of treating camera poses as discrete entities, we model the entire flight trajectory as a continuous, differentiable function $\phi$. This function maps a timestamp $t$ to a 6-DoF pose (translation $\hat{x}_i$ and rotation $\hat{q}_i$). The architecture employs a 1-D multi-resolution hash grid to capture high-frequency trajectory details, followed by a small Multi-Layer Perceptron (MLP).
As shown in Figure 6, the MLP consists of several layers with 1024 units and a skip connection to facilitate gradient flow. This continuous representation allows the system to interpolate precise poses for depth maps captured between RGB frames.
2. Bootstrapping and Initialization
The training process begins with a bootstrapping phase. First, the time-pose function is fitted to initial poses obtained via Structure-from-Motion (SfM) from the RGB sequence. Simultaneously, a large-scale NeRF—based on the Mega-NeRF architecture to handle city-scale scenes—is initialized using only the RGB data.
3. Joint Optimization
The core of our approach is the joint optimization of the NeRF parameters $\theta$ and the time-pose function parameters $\phi$. When a depth map is encountered at timestamp $t_j$, the function $\phi(t_j)$ predicts the exact pose. The resulting depth loss provides gradients that simultaneously refine the 3D scene geometry and "correct" the trajectory function. To ensure physical plausibility, a Speed Loss is incorporated, regularizing the trajectory based on the drone's average linear speed.
Experiments and Results
We evaluated AsyncNeRF on both synthetic and real-world datasets. The Asynchronous Urban Scene (AUS) dataset was created using the AirSim simulator to provide photorealistic city models with controlled temporal offsets. Real-world validation was conducted using data from a DJI M300 UAV equipped with LiDAR and a high-definition camera.
Quantitative Performance
In large-scale urban reconstruction tasks, AsyncNeRF achieved a 35.26% reduction in Root Mean Square Error (RMSE) compared to state-of-the-art baselines like Mega-NeRF. Furthermore, the joint optimization stage significantly improved pose accuracy, reducing translation errors from approximately 1.07m to 0.53m.
Qualitative Analysis
Qualitatively, the method produces significantly sharper depth maps and eliminates "floaters"—common artifacts in empty space that occur when depth supervision is absent or misaligned. AsyncNeRF demonstrates superior performance in rendering complex urban geometry compared to NeRF-W and standard Mega-NeRF.
Conclusion
AsyncNeRF effectively addresses the challenge of asynchronous RGB-D data in large-scale 3D reconstruction. By leveraging a continuous trajectory prior and joint optimization, the framework achieves self-alignment of sensors, leading to substantial improvements in both geometric accuracy and visual quality. While the current model assumes a static scene and depends on initial SfM poses, it sets a new benchmark for robust UAV-based mapping.
Summary updated