Abstract
PaReNeRF is a novel framework designed to accelerate high-fidelity 3D reconstruction of large-scale dynamic scenes, such as urban environments. By rendering low-resolution feature maps and utilizing a specialized reference decoder to "borrow" high-frequency details from adjacent video frames, the system reduces training time by 60% and inference time by 87%. Experimental results on the KITTI dataset demonstrate that PaReNeRF not only improves speed but also enhances visual quality, achieving a 13% improvement in PSNR over previous state-of-the-art methods like SUDS.
Introduction
Creating digital twins of real-world driving environments is essential for autonomous vehicle simulation. Neural Radiance Fields (NeRFs) have become the standard for this task; however, they face a significant "speed vs. quality" trade-off in large-scale dynamic settings. Current state-of-the-art models, such as SUDS, can require over two days to train a single 9-second video clip, with rendering times per frame reaching several seconds. While lower-resolution rendering can improve speed, it often results in blurry images that lose critical details like street signs or license plates. PaReNeRF addresses these challenges by achieving high-speed rendering without sacrificing photo-realism.
Methodology
The PaReNeRF framework introduces three primary innovations to the standard NeRF pipeline to break the speed-quality deadlock.
Patch-based Sampling and Feature Rendering
Instead of sampling individual rays, the model samples small $16 \times 16$ patches. This approach increases data throughput during training and enables the use of Convolutional Neural Networks (CNNs) for image-level processing. To further accelerate the process, PaReNeRF renders a low-resolution (LR) "feature map" rather than full-resolution RGB values, reducing expensive ray-marching queries by nearly 10x.
Reference Decoder with Prior Search
To restore details lost during LR rendering, a lightweight reference decoder is employed. The system utilizes optical flow to track patch movement between frames and Structural Similarity (SSIM) to identify the best-matching high-resolution (HR) "reference patch" from adjacent views. This reference patch is fused with the rendered feature map using a Cascading Residual Network (CARN) to produce the final high-detail image.
Experimental Results
The model was evaluated on the KITTI and Virtual KITTI 2 (VKITTI2) benchmarks. Compared to SUDS, PaReNeRF demonstrated significant performance gains:
- Speed: Training time was reduced from 64 hours to 26 hours (a 60% improvement), and inference time dropped from 12.86s to 1.61s per frame (an 87% improvement).
- Quality: The model achieved a PSNR of 32.6, representing a 13% improvement over SUDS (28.3).
Qualitative analysis shows that PaReNeRF successfully reconstructs sharp details on traffic signs and vehicle textures, effectively eliminating the "ghosting" artifacts and blur prevalent in previous methods.
Conclusion
PaReNeRF represents a significant advancement toward real-time, high-fidelity 3D reconstruction for autonomous driving simulations. By exploiting temporal redundancy and rendering in feature space, it overcomes the computational bottlenecks of traditional NeRFs. While current inference speeds (1.6s per frame) are not yet sufficient for 30+ FPS interactive simulation, the framework provides a robust foundation for future optimizations in large-scale dynamic scene modeling.
Summary updated