AsyncNeRF: Learning Large-scale Radiance Fields from Asynchronous RGB-D Sequences with Time-Pose Function

Zirui Wu, Yuantao Chen, Runyi Yang, Zhenxin Zhu, Chao Hou, Yongliang Shi, Hao Zhao, Guyue Zhou

Abstract

Large-scale radiance fields are promising mapping tools for smart transportation applications like autonomous driving or drone delivery. However, for large-scale scenes, compact synchronized RGB-D cameras are often inapplicable due to limited sensing range, and using separate RGB and depth sensors inevitably leads to unsynchronized sequences. We propose AsyncNeRF, the first solution that self-calibrates the mismatch between RGB and depth frames. By modeling the camera trajectory as a continuous time-pose function, we enable joint optimization of the scene representation and sensor poses. Benchmarking on a diverse, photorealistic dataset demonstrates that AsyncNeRF significantly outperforms prior counterparts in both reconstruction quality and geometric accuracy.

The Challenge of Asynchronous RGB-D Mapping

Incorporating depth data into Neural Radiance Fields (NeRFs) is a proven way to speed up convergence and eliminate "floaters" (artifacts in empty space). However, existing methods assume that RGB images and depth maps are perfectly synchronized. In real-world scenarios, especially with Unmanned Aerial Vehicles (UAVs) flying at high speeds, hardware or software delays often lead to temporal asynchrony. If a depth map is captured even a few milliseconds after its corresponding RGB image, the camera has already moved. Using these misaligned pairs for supervision leads to blurry reconstructions and geometric errors.

Conceptual illustration of the asynchronous RGB-D problem (a) and the proposed continuous time-pose function solution (b).

Methodology: The Time-Pose Function

The core innovation of AsyncNeRF is the Time-Pose Function ($\phi$), an implicit neural network that maps a single timestamp ($t$) to a 6-DoF camera pose ($SE(3)$). This approach leverages the domain-specific fact that while sensors may be asynchronous, they are sampled from the same continuous physical trajectory.

The optimization pipeline consists of three stages:

  1. Time-Pose Function Fitting: The model learns the drone's trajectory by fitting the function to initial poses obtained from Structure-from-Motion (SfM) on the RGB sequence.
  2. Radiance Field Bootstrapping: A large-scale NeRF is initialized using RGB images and the poses predicted by the time-pose function.
  3. Joint Optimization: The system jointly optimizes the scene representation and the time-pose function. Because the pipeline is fully differentiable, depth maps provide supervision at their exact capture timestamps, "nudging" the trajectory to align perfectly with observed data.

Experimental Validation

To validate the approach, the authors introduced the Asynchronous Urban Scene (AUS) Dataset, a large-scale synthetic benchmark built in AirSim. It includes six diverse environments (e.g., New York, San Francisco) and various trajectory types. Asynchrony was simulated using both fixed offsets and random temporal jitter. Real-world validation was also performed using a DJI M300 UAV equipped with high-definition cameras and LiDAR.

Key Results and Performance

AsyncNeRF significantly outperformed baselines like Mega-NeRF and NeRF-W on the AUS dataset across all metrics:

  • Image Quality: Achieved a +1.11 dB improvement in PSNR over Mega-NeRF.
  • Geometry: The Root Mean Square Error (RMSE) for depth dropped from 46.52 to 11.26, a ~75% reduction in error.
  • Pose Accuracy: Joint optimization cut depth sensor pose error in half, reducing translation error from 1.07m to 0.53m.
Overview of AsyncNeRF: (a-b) problem and solution diagrams, (c) input RGB, (d) baseline depth reconstruction with misalignment, and (e) AsyncNeRF's sharp, aligned depth reconstruction.

Qualitatively, the method produces much sharper depth maps and avoids the "smearing" artifacts typically seen in glass or thin structures when using misaligned data. By treating the trajectory as a continuous function rather than discrete points, AsyncNeRF provides a robust solution for high-speed robotic mapping.

Summary updated

Abstract

Large-scale radiance fields are promising mapping tools for smart transportation applications like autonomous driving or drone delivery. But for large-scale scenes, compact synchronized RGB-D cameras are not applicable due to limited sensing range, and using separate RGB and depth sensors inevitably leads to unsynchronized sequences. Inspired by the recent success of self-calibrating radiance field training methods that do not require known intrinsic or extrinsic parameters, we propose the first solution that self-calibrates the mismatch between RGB and depth frames. We leverage the important domain-specific fact that RGB and depth frames are actually sampled from the same trajectory and develop a novel implicit network called the time-pose function. Combining it with a large-scale radiance field leads to an architecture that cascades two implicit representation networks. To validate its effectiveness, we construct a diverse and photorealistic dataset that covers various RGB-D mismatch scenarios. Through a comprehensive benchmarking on this dataset, we demonstrate the flexibility of our method in different scenarios and superior performance over applicable prior counterparts. Codes, data, and models will be made publicly available.
BibTeX
@inproceedings{wu2025asyncnerf,
  author = {Zirui Wu and Yuantao Chen and Runyi Yang and Zhenxin Zhu and Chao Hou and Yongliang Shi and Hao Zhao and Guyue Zhou},
  title = {{AsyncNeRF: Learning Large-scale Radiance Fields from Asynchronous RGB-D Sequences with Time-Pose Function}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00061},
}
Figure