TL;DR
This paper introduces Deblur-NSFF, a novel method for generating sharp novel views and time-synthesized videos of dynamic scenes, even when the input footage is blurry. By learning spatially-varying blur kernels and a time-conditioned NeRF, Deblur-NSFF effectively disentangles blur from scene motion, outperforming existing methods in sharpness and detail recovery.
What Problem Does This Paper Address?
The field of novel view synthesis, particularly for dynamic scenes, has seen remarkable progress with Neural Radiance Fields (NeRFs) and their extensions like Neural Scene Flow Fields (NSFF). These methods excel at reconstructing detailed 3D scenes and their temporal evolution, enabling applications like free-viewpoint video. However, a significant practical limitation is their sensitivity to input image quality. Standard NeRF-based approaches assume well-captured, blur-free input, which is rarely the case in real-world scenarios due to camera motion, object movement, or out-of-focus blur. This blurriness degrades the quality of synthesized views, making existing methods unsuitable for many practical applications. While some methods like Deblur-NeRF have addressed blur in static scenes, handling blur in dynamic scenes remains an open challenge.
Methods
Deblur-NSFF builds upon the Neural Scene Flow Fields (NSFF) framework, which models dynamic scenes by learning a time-independent static NeRF and a time-dependent dynamic NeRF. The core innovation of Deblur-NSFF lies in its ability to learn and simulate the blurring process.
- Spatially-Varying Blur Kernel Estimation: The method introduces a Kernel Estimation Network (KernelNet) that learns spatially-varying blur kernels for each pixel. This network takes a pixel's location, a canonical kernel, and a view embedding as input to predict kernel offsets and weights. This allows for modeling complex, non-uniform blurring.
- Time-Conditioned NeRF: A sharp, time-conditioned NeRF is trained to represent the scene. This NeRF learns to output reflectance, density, and 3D scene motion.
- Blur Simulation and Loss Functions: During training, the synthesized sharp scene is convolved with the predicted blur kernels to simulate the blurry input. This simulated blur is then used in a loss function (Lmse-blur) to supervise the KernelNet. Additionally, temporal photometric consistency losses (Lpho) are employed to ensure consistency across time, warping views using predicted scene flow and applying the learned blur kernel. An alignment loss (Lalign) is also used to regularize the kernel estimation and ensure the NeRF learns a sharp representation.
- Inference: For inference, the KernelNet is discarded, and the trained sharp NeRF is used to generate novel space-time views directly.
Datasets
The primary dataset used for training and evaluation is the NVIDIA Dynamic Scenes Dataset [35]. The authors modified this dataset by introducing synthetic Gaussian blur to a portion of the frames for training.
Key Insights and Ideas
- Disentangling Blur and Motion: The most crucial insight is that by explicitly learning and simulating the blur kernel, the underlying NeRF can be trained to represent a sharp, underlying scene, effectively separating the blur artifact from the scene's inherent motion.
- Spatially-Varying Kernels are Essential: Real-world blur is rarely uniform. The use of spatially-varying blur kernels is key to accurately modeling and removing blur across different parts of an image and scene.
- Leveraging NSFF for Dynamic Scenes: Building upon the NSFF framework provides a strong foundation for handling dynamic scenes, allowing Deblur-NSFF to inherit its capabilities for motion modeling while adding blur robustness.
- Practicality for Real-World Data: The method directly addresses the limitations of current NeRFs when faced with imperfect real-world input, paving the way for more robust applications.
Key Results
- Quantitative Performance: Deblur-NSFF significantly outperforms prior methods like DVS [5] and NSFF [8] in terms of PSNR, SSIM, and LPIPS metrics for both full scene and dynamic region synthesis. For example, compared to NSFF, Deblur-NSFF achieves a 4.3% improvement in PSNR and a 12.8% improvement in SSIM for the full scene.
- Qualitative Performance: Visual comparisons demonstrate that Deblur-NSFF produces significantly sharper renderings, recovering finer details in both static and dynamic regions. It effectively sharpens backgrounds and resolves artifacts that plague other methods when dealing with blurry inputs.
- Comparison with Deblur-NeRF: While Deblur-NeRF [14] shows promise for static scenes, Deblur-NSFF demonstrates superior performance on dynamic scenes, recovering sharper details in dynamic regions where Deblur-NeRF exhibits artifacts.
Limitations and Future Work
- Minute Detail Recovery: The paper acknowledges that the method struggles to recover very fine details, such as facial expressions, as shown in Figure 6a.
- Time Interpolation Artifacts: While space interpolation is generally good, time interpolation or space-time interpolation can sometimes produce artifacts (Figure 6b). The authors attribute this to the splatting-based plane-sweep volume rendering approach used for time interpolation.
- Future Work: The authors suggest that improving time interpolation and exploring more advanced rendering techniques could further enhance the method's capabilities.
Summary updated