From NeRFLiX to NeRFLiX++: A General NeRF-Agnostic Restorer Paradigm

Kun Zhou, Wenbo Li, Nianjuan Jiang, Xiaoguang Han, Jiangbo Lu

TL;DR

NeRFLiX++ is a general, "NeRF-agnostic" post-processing framework designed to remove rendering artifacts like noise and blur from any Neural Radiance Field (NeRF) model. By combining a sophisticated two-stage degradation simulator with a high-efficiency inter-viewpoint mixer, it can restore photo-realistic 4K details from noisy, low-resolution NeRF outputs up to 9 times faster than its predecessor.

What Problem Does This Paper Address?

While Neural Radiance Fields (NeRFs) have revolutionized novel view synthesis, they are rarely perfect. Even with high-quality training data, the final rendered views often suffer from "rendering artifacts"—a specific mix of noise and blur caused by inaccurate camera calibration, simplified camera models, or scene representation errors.

Existing high-resolution NeRF variants often require scene-specific, multi-stage training that is computationally expensive and tightly coupled to a specific NeRF architecture. The authors seek a general restorer that can be applied to any NeRF model without retraining the original scene representation, effectively decoupling the rendering and refinement stages.

Fig. 2: A general illustration of NeRF-based novel view syn- thesis pipeline. Three main steps are involved: (1) ray shooting, (2) ray marching, and (3) radiance accumulation.

Methods

The authors propose a paradigm shift from NeRFLiX to NeRFLiX++, focusing on two main pillars:

  1. Two-Stage Degradation Modeling: To train a restorer, you need paired "clean" and "noisy" data. Since real NeRF-rendered pairs are scarce, the authors simulate them.
  2. Stage 1 (Manual): Uses hand-crafted degradations like "splatted Gaussian noise" (which mimics how rays hitting a 3D point are re-projected into a 2D area), ray jittering, and anisotropic blur. NeRFLiX++ adds illumination jetting and lightness compression to increase realism.
  3. Stage 2 (Deep Generative): A GAN-based simulator refines the manual output to bridge the statistical "domain gap" between simulated noise and real NeRF artifacts. They introduce a Weighted Top-K Buddy Loss (WKS), which uses image self-similarity to provide diverse supervisory signals even when target data is limited.
  4. Guided Inter-Viewpoint Mixer (G-IVM): This is the "engine" that restores the image by fusing information from high-quality reference views.
  5. Unlike the original NeRFLiX which used a slow recurrent strategy, G-IVM uses a coarse-to-fine approach.
  6. It estimates optical flow at a low resolution to guide the aggregation of features at higher scales. This significantly reduces the computational burden, making 4K restoration feasible.

Datasets

The models were trained and evaluated using several standard benchmarks:

  • Vimeo90K: Used for large-scale raw video sequences to generate simulated training pairs.
  • LLFF (Local Light Field Fusion): A standard dataset for forward-facing real-world scenes.
  • Tanks and Temples: Real-world scenes with large viewpoint variations.
  • Noisy LLFF Synthetic: A virtual dataset where camera "jetting" was manually applied to simulate imperfect real-world calibration.
  • BungeeNeRF: Used to test generalization on city-scale, multi-scale scenes.

Key Insights and Ideas

  • NeRF Noise is Unique: Standard image denoising doesn't work well because NeRF artifacts are "splatted"—they are spatially variant and tied to the 3D geometry and camera poses.
  • Decoupling is Key: By making the restorer NeRF-agnostic, it can be plugged into any future NeRF architecture. It treats the NeRF as a "noisy renderer" and uses the original high-quality training photos as a ground-truth reference for fusion.
  • Training Acceleration: A fascinating finding is that using NeRFLiX++ allows researchers to train their base NeRF models for 50% less time. The restorer can fix the artifacts resulting from under-training, often producing better final results than a full-time trained NeRF alone.
  • Efficiency via Downsampling: Estimating correspondences (optical flow) at a lower resolution is not just faster; it's actually more accurate for NeRF restoration because downsampling reduces the displacement between views and narrows the distribution gap between rendered and reference images.

Key Results

NeRFLiX++ demonstrates state-of-the-art performance across multiple metrics (PSNR, SSIM, LPIPS):

  • Speed: It is 9.2× faster than the original NeRFLiX for 1024x1024 images and can process 2048x2048 frames in just 1.5 seconds.
  • Quality Boost: On the LLFF dataset, it improved the PSNR of models like TensoRF and Plenoxels by over 0.6 dB.
  • 4K Restoration: It successfully generates photo-realistic 4K frames from noisy 1K NeRF inputs, outperforming specialized models like 4K-NeRF.
Fig. 1: Visualization of restoration results of our proposed NeRFLiX++ for 4K images. It is clear that NeRFLiX++ produces photo-realistic 4K frames from low-resolution and noisy inputs rendered by DVGO [48].

Limitations and Future Work

  • Reference Dependency: The system relies on having high-quality reference views that overlap with the target novel view. In scenes with extremely sparse input views, the restoration quality may degrade.
  • Flow Accuracy: While the guided mixer is efficient, it still relies on optical flow. In areas with complex occlusions or very thin structures, flow estimation can still be a bottleneck.
  • Future Directions: The authors suggest exploring even more efficient aggregation modules and extending the paradigm to dynamic scenes where temporal consistency is even more critical.

Summary updated

Abstract

Neural radiance fields (NeRF) have shown great success in novel view synthesis. However, recovering high-quality details from real-world scenes is still challenging for the existing NeRF-based approaches, due to the potential imperfect calibration information and scene representation inaccuracy. Even with high-quality training frames, the synthetic novel views produced by NeRF models still suffer from notable rendering artifacts, such as noise and blur. To address this, we propose NeRFLiX, a general NeRF-agnostic restorer paradigm that learns a degradation-driven inter-viewpoint mixer. Specially, we design a NeRF-style degradation modeling approach and construct large-scale training data, enabling the possibility of effectively removing NeRF-native rendering artifacts for deep neural networks. Moreover, beyond the degradation removal, we propose an inter-viewpoint aggregation framework that fuses highly related high-quality training images, pushing the performance of cutting-edge NeRF models to entirely new levels and producing highly photo-realistic synthetic views. Based on this paradigm, we further present NeRFLiX++ with a stronger two-stage NeRF degradation simulator and a faster inter-viewpoint mixer, achieving superior performance with significantly improved computational efficiency. Notably, NeRFLiX++ is capable of restoring photo-realistic ultra-high-resolution outputs from noisy low-resolution NeRF-rendered views. Extensive experiments demonstrate the excellent restoration ability of NeRFLiX++ on various novel view synthesis benchmarks.
BibTeX
@article{zhou2024from,
  author = {Kun Zhou and Wenbo Li and Nianjuan Jiang and Xiaoguang Han and Jiangbo Lu},
  title = {{From NeRFLiX to NeRFLiX++: A General NeRF-Agnostic Restorer Paradigm}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2024},
  url = {https://openreview.net/forum?id=UjCmtUuwHf},
}
Figure