Self-Aligning Depth-Regularized Radiance Fields for Asynchronous RGB-D Sequences

Yuxin Huang, Andong Yang, Yuantao Chen, Runyi Yang, Zhenxin Zhu, Chao Hou, Hao Zhao, Guyue Zhou

Introduction

Neural Radiance Fields (NeRFs) have revolutionized 3D scene reconstruction by enabling high-fidelity novel view synthesis from sparse image sets. Recent research has demonstrated that incorporating depth information (RGB-D) significantly enhances reconstruction quality and accelerates convergence by providing explicit geometric constraints. However, a significant practical challenge remains: sensor asynchrony. In real-world applications, particularly those involving Unmanned Aerial Vehicles (UAVs), RGB cameras and depth sensors (e.g., LiDAR) often operate at different frequencies due to hardware and power constraints.

Overview of the asynchrony problem (a), the proposed continuous trajectory solution (b), and visual examples (c-e).

As illustrated in Figure 3(a), this temporal mismatch means that the camera's pose at timestamp $t_{rgb}$ does not coincide with the sensor's pose at $t_{depth}$. Directly supervising a NeRF with misaligned depth maps leads to geometric blurring and artifacts. To address this, we introduce AsyncNeRF, a framework that models the drone's trajectory as a continuous function to self-align asynchronous data during training.

Methodology

The AsyncNeRF pipeline consists of three primary components designed to bridge the temporal gap between sensors.

1. Continuous Time-Pose Function

Instead of treating camera poses as discrete entities, we model the entire flight trajectory as a continuous, differentiable function $\phi$. This function maps a timestamp $t$ to a 6-DoF pose (translation $\hat{x}_i$ and rotation $\hat{q}_i$). The architecture employs a 1-D multi-resolution hash grid to capture high-frequency trajectory details, followed by a small Multi-Layer Perceptron (MLP).

Architecture of the time-pose function MLP with skip connections used to predict poses from timestamps.

As shown in Figure 6, the MLP consists of several layers with 1024 units and a skip connection to facilitate gradient flow. This continuous representation allows the system to interpolate precise poses for depth maps captured between RGB frames.

2. Bootstrapping and Initialization

The training process begins with a bootstrapping phase. First, the time-pose function is fitted to initial poses obtained via Structure-from-Motion (SfM) from the RGB sequence. Simultaneously, a large-scale NeRF—based on the Mega-NeRF architecture to handle city-scale scenes—is initialized using only the RGB data.

3. Joint Optimization

The core of our approach is the joint optimization of the NeRF parameters $\theta$ and the time-pose function parameters $\phi$. When a depth map is encountered at timestamp $t_j$, the function $\phi(t_j)$ predicts the exact pose. The resulting depth loss provides gradients that simultaneously refine the 3D scene geometry and "correct" the trajectory function. To ensure physical plausibility, a Speed Loss is incorporated, regularizing the trajectory based on the drone's average linear speed.

Experiments and Results

We evaluated AsyncNeRF on both synthetic and real-world datasets. The Asynchronous Urban Scene (AUS) dataset was created using the AirSim simulator to provide photorealistic city models with controlled temporal offsets. Real-world validation was conducted using data from a DJI M300 UAV equipped with LiDAR and a high-definition camera.

Quantitative Performance

In large-scale urban reconstruction tasks, AsyncNeRF achieved a 35.26% reduction in Root Mean Square Error (RMSE) compared to state-of-the-art baselines like Mega-NeRF. Furthermore, the joint optimization stage significantly improved pose accuracy, reducing translation errors from approximately 1.07m to 0.53m.

Qualitative Analysis

Qualitatively, the method produces significantly sharper depth maps and eliminates "floaters"—common artifacts in empty space that occur when depth supervision is absent or misaligned. AsyncNeRF demonstrates superior performance in rendering complex urban geometry compared to NeRF-W and standard Mega-NeRF.

Conclusion

AsyncNeRF effectively addresses the challenge of asynchronous RGB-D data in large-scale 3D reconstruction. By leveraging a continuous trajectory prior and joint optimization, the framework achieves self-alignment of sensors, leading to substantial improvements in both geometric accuracy and visual quality. While the current model assumes a static scene and depends on initial SfM poses, it sets a new benchmark for robust UAV-based mapping.

Summary updated

Abstract

It has been shown that learning radiance fields with depth rendering and depth supervision can effectively promote the quality and convergence of view synthesis. However, this paradigm requires input RGB-D sequences to be synchronized. In the UAV city modeling scenario, there exists asynchrony between RGB images and depth images due to the different frequencies of the solid-state LiDAR and RGB sensors. To synthesize high-quality views in such a scenario, we propose a novel time-pose function, which is an implicit network that maps timestamps to SE(3) elements. To train this function, we also design a joint optimization scheme to jointly learn the large-scale depth-regularized radiance fields and the time-pose function. Furthermore, we propose a large synthetic dataset with diverse controlled mismatches and ground truth to evaluate this new problem setting systematically. The proposed approach has been evaluated on both datasets and in a real drone. To evaluate the impact of view density, each algorithm was test on three different trajectories with different view densities. Compared to state-of-the-art baseline methods, the proposed approach reduces reconstruction error by 35.26% in city modeling scenarios. Our code is available at github.com/saythe17/AsyncNeRF.
BibTeX
@inproceedings{huang2025self,
  author = {Yuxin Huang and Andong Yang and Yuantao Chen and Runyi Yang and Zhenxin Zhu and Chao Hou and Hao Zhao and Guyue Zhou},
  title = {{Self-Aligning Depth-Regularized Radiance Fields for Asynchronous RGB-D Sequences}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00061},
}
Figure