Ced-NeRF: A Compact and Efficient Method for Dynamic Neural Radiance Fields

Youtian LinAAAI 2024 · OpenReview

Abstract

Rendering photorealistic dynamic scenes has been a focus of recent research, with applications in virtual and augmented reality. While the Neural Radiance Field (NeRF) has shown remarkable rendering quality for static scenes, achieving real-time rendering of dynamic scenes remains challenging due to expansive computation for the time dimension. The incorporation of explicit-based methods, specifically voxel grids, has been proposed to accelerate the training and rendering of neural radiance fields with hybrid representation. However, employing a hybrid representation for dynamic scenes results in overfitting due to fast convergence, which can result in artifacts (e.g., floaters, noisy geometric) on novel views. To address this, we propose a compact and efficient method for dynamic neural radiance fields, namely Ced-NeRF which only require a small number of additional parameters to construct a hybrid representation of dynamic NeRF. Evaluation of dynamic scene datasets shows that our Ced-NeRF achieves fast rendering speeds while maintaining high-quality rendering results. Our method outperforms the current state-of-the-art methods in terms of quality, training and rendering speed.

What Problem Does This Paper Address?

Neural Radiance Fields (NeRFs) have revolutionized how we synthesize novel views of static scenes, but dynamic scenes—where objects move, deform, or change lighting—remain a massive computational hurdle.

To speed things up, researchers often use hybrid representations (like voxel grids or hash tables) which store features explicitly. While these are lightning-fast for static scenes, they often fail on dynamic ones. Because these grids converge so quickly, they tend to "overfit" to the early, inaccurate guesses of the deformation network. This results in visual "floaters," noisy geometry, and a failure to render clean views from new angles.

Comparison between training and test views highlighting artifacts in novel view synthesis.

Ced-NeRF aims to keep the speed of hybrid grids while fixing these "dirty" reconstruction artifacts.

Methods

The authors introduce three key innovations to stabilize the training of dynamic hybrid NeRFs:

  1. Hierarchical Motion Prediction: Instead of asking a single network to predict a complex 3D warp, Ced-NeRF splits the task. It predicts a coarse motion (moving between grid cells) and a fine-grained motion (moving within a single cell). By bounding the fine-grained movement with a tanh function and a specific step size, the model better decouples global movement from local detail.
  2. Neural Latent Regularization: This is the "secret sauce" for preventing overfitting. The authors add a tiny auxiliary MLP ($\Phi_f$) that tries to reconstruct the grid features from coordinates and time. This acts as a global regularizer, forcing the grid to evolve toward a "mean" representation rather than jumping to noisy, local solutions early in training. Crucially, this MLP is discarded after training, so it doesn't slow down rendering.
  3. Attenuating Time Feature: Usually, feeding "time" directly into a NeRF helps model lighting changes but can confuse the model into using time as a crutch for motion. Ced-NeRF attenuates the time signal based on the magnitude of deformation—essentially telling the model: "If something is moving a lot, rely on geometry; if it's still, you can use the time feature to handle lighting shifts."

Key Results

Ced-NeRF achieves state-of-the-art (SOTA) results while being significantly faster than previous methods across standard benchmarks like D-NeRF, HyperNeRF, and DyNeRF.

Ced-NeRF performance on D-NeRF scenes, showing high quality (34.2 dB) achieved in just 4 minutes.
  • Training Speed: On the D-NeRF dataset, it trains in just 4 minutes, compared to 20 hours for the original D-NeRF and 30 minutes for TiNeuVox.
  • Rendering Speed: It renders at approximately 0.16 seconds per image, making it orders of magnitude faster than traditional implicit methods like HyperNeRF (~90s per image).
  • Quality: On the D-NeRF benchmark, it achieved a PSNR of 34.21 dB, outperforming the previous SOTA (TiNeuVox at 32.67 dB) while using half the training time.
  • Visual Fidelity: Qualitative results show a marked reduction in "floaters" and much sharper edges on deforming objects compared to K-Planes or D-NeRF.

Conclusion

Ced-NeRF demonstrates that the speed of hybrid representations can be harnessed for dynamic scenes without sacrificing visual quality. By identifying the "Speed-Accuracy Paradox"—where fast convergence leads to geometric errors—and introducing latent regularization, the authors provide a robust framework for efficient 4D scene reconstruction. While future work may focus on even faster training or handling extremely subtle deformations, Ced-NeRF represents a significant step toward real-time dynamic view synthesis.

Summary updated

Abstract

Rendering photorealistic dynamic scenes has been a focus of recent research, with applications in virtual and augmented reality. While the Neural Radiance Field (NeRF) has shown remarkable rendering quality for static scenes, achieving real-time rendering of dynamic scenes remains challenging due to expansive computation for the time dimension. The incorporation of explicit-based methods, specifically voxel grids, has been proposed to accelerate the training and rendering of neural radiance fields with hybrid representation. However, employing a hybrid representation for dynamic scenes results in overfitting due to fast convergence, which can result in artifacts (e.g., floaters, noisy geometric) on novel views. To address this, we propose a compact and efficient method for dynamic neural radiance fields, namely Ced-NeRF which only require a small number of additional parameters to construct a hybrid representation of dynamic NeRF. Evaluation of dynamic scene datasets shows that our Ced-NeRF achieves fast rendering speeds while maintaining high-quality rendering results. Our method outperforms the current state-of-the-art methods in terms of quality, training and rendering speed.
BibTeX
@inproceedings{lin2024ced,
  author = {Youtian Lin},
  title = {{Ced-NeRF: A Compact and Efficient Method for Dynamic Neural Radiance Fields}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=q8IvfDtSPr},
}
Figure