Self-Aligning Depth-regularized Radiance Fields for Asynchronous RGB-D Sequences

Yuxin Huang, Andong Yang, Zirui Wu, Yuantao Chen, Runyi Yang, Zhenxin Zhu, Chao Hou, Hao Zhao, Guyue Zhou

TL;DR

This paper introduces a method to train depth-regularized Neural Radiance Fields (NeRFs) using asynchronous RGB-D sequences, a common challenge in high-speed drone (UAV) mapping. By modeling the camera trajectory as a continuous, differentiable "time-pose function," the authors can align misaligned sensor data and achieve significantly higher quality 3D reconstructions and depth estimations than existing state-of-the-art methods.

What Problem Does This Paper Address?

Neural Radiance Fields (NeRFs) have revolutionized 3D scene reconstruction, and recent work shows that adding depth information (like from LiDAR or depth cameras) significantly speeds up training and improves accuracy. However, there's a catch: most methods assume the RGB images and depth maps are perfectly synchronized in time.

In real-world scenarios, especially with Unmanned Aerial Vehicles (UAVs) flying at high speeds, this synchronization is hard to maintain. Sensors might capture data at different rates or with slight temporal offsets. This asynchrony leads to "ghosting" or blurred geometry because the depth map for a specific timestamp doesn't perfectly match the camera's perspective in the corresponding RGB image.

The problem of asynchrony: RGB and Depth (D) frames are captured at different timestamps ($t_1, t_2, ...$), making direct supervision difficult.

Methods

The authors propose a three-stage optimization pipeline to bridge the gap between asynchronous sensors:

  1. Time-Pose Function Fitting: Instead of treating camera poses as discrete points, they model the entire drone trajectory as a continuous function $\phi(t) \to \text{SE}(3)$. This function is an implicit neural network using a 1D multi-resolution hash grid (similar to Instant-NGP but for time). It learns to map any timestamp to a 6-degree-of-freedom camera pose.
  2. Radiance Field Bootstrapping: They initialize a large-scale radiance field (based on Mega-NeRF) using only the RGB images and their initial poses (often derived from Structure-from-Motion).
  3. Joint Optimization: This is the "secret sauce." Because both the radiance field ($\theta$) and the time-pose function ($\phi$) are differentiable, they can be optimized simultaneously. The system uses the time-pose function to "look up" the correct pose for a depth map's timestamp, then uses that depth to regularize the radiance field. Conversely, the radiance field's errors help refine the trajectory function.
The 3-step optimization pipeline: (i) fitting the time-pose function, (ii) bootstrapping the NeRF, and (iii) joint optimization.

Datasets

The authors evaluated their approach on two main fronts:

  • Asynchronous Urban Scene (AUS) Dataset: A new, photo-realistic synthetic dataset created using the AirSim simulator. It includes six large-scale city and cultural heritage models with various drone trajectories (simple zig-zags to complex manual flight) and controlled temporal offsets between RGB and depth data.
  • Real-world Drone Data: Captured using a DJI M300 UAV equipped with a high-def RGB camera and a LiDAR sensor.

Key Insights and Ideas

  • Trajectory Prior: The core insight is that while sensors are asynchronous, they are physically attached to the same drone. Therefore, they must follow the same underlying continuous physical trajectory.
  • Implicit Trajectory Representation: Using a 1D hash grid for the time-pose function allows the model to capture both smooth, coarse movements and high-frequency jitters in the drone's flight path.
  • Differentiable Alignment: By making the alignment process part of the neural network's loss function, the system doesn't need perfect hardware synchronization; it "learns" the correct alignment through backpropagation.

Key Results

The proposed method significantly outperforms baselines like Mega-NeRF and NeRF-W, especially in geometric accuracy:

  • View Synthesis: On the AUS dataset, the method achieved a 1.11 dB improvement in PSNR over Mega-NeRF.
  • Depth Accuracy: The most dramatic gains were in depth estimation, with a reduction in Root Mean Square Error (RMSE) by 35.26 compared to Mega-NeRF.
  • Pose Refinement: The joint optimization stage cut the initial pose errors in half, reducing translation error from 1.07m to 0.53m and rotation error from 1.04° to 0.41°.

Qualitatively, the method produces much sharper depth maps and cleaner novel view renderings, successfully handling tricky areas like glass windows and fine architectural details that typically fail in asynchronous setups.

Qualitative comparison showing that the proposed method (middle row) produces significantly more accurate depth maps and sharper novel views than Mega-NeRF or standard NeRF.

Limitations and Future Work

While highly effective, the authors note a few areas for improvement:

  • Imagery Features: Currently, the time-pose function relies on geometric and temporal priors. Incorporating visual features (like optical flow or feature matching) directly into the trajectory estimation could make it even more robust.
  • Dataset Expansion: While the AUS dataset is a great start, the authors plan to add even more diverse scenes and environmental conditions to further stress-test the system.
  • Computational Cost: Jointly optimizing two implicit networks is computationally intensive, suggesting a need for further optimization to reach real-time or near-real-time performance.

Summary updated

Abstract

It has been shown that learning radiance fields with depth rendering and depth supervision can effectively promote the quality and convergence of view synthesis. However, this paradigm requires input RGB-D sequences to be synchronized, hindering its usage in the UAV city modeling scenario. As there exists asynchrony between RGB images and depth images due to high-speed flight, we propose a novel time-pose function, which is an implicit network that maps timestamps to $\rm SE(3)$ elements. To simplify the training process, we also design a joint optimization scheme to jointly learn the large-scale depth-regularized radiance fields and the time-pose function. Our algorithm consists of three steps: (1) time-pose function fitting, (2) radiance field bootstrapping, (3) joint pose error compensation and radiance field refinement. In addition, we propose a large synthetic dataset with diverse controlled mismatches and ground truth to evaluate this new problem setting systematically. Through extensive experiments, we demonstrate that our method outperforms baselines without regularization. We also show qualitatively improved results on a real-world asynchronous RGB-D sequence captured by drone. Codes, data, and models will be made publicly available.
BibTeX
@inproceedings{huang2025self,
  author = {Yuxin Huang and Andong Yang and Zirui Wu and Yuantao Chen and Runyi Yang and Zhenxin Zhu and Chao Hou and Hao Zhao and Guyue Zhou},
  title = {{Self-Aligning Depth-regularized Radiance Fields for Asynchronous RGB-D Sequences}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00061},
}
Figure