Separating Direct and Global Components from Novel Viewpoints

Kengo Matsufuji, Lin Shi, Ryo Kawahara, Takahiro OkabeWACV 2025

TL;DR

This paper introduces a novel method to disentangle direct (e.g., diffuse, specular) and global (e.g., inter-reflection, subsurface scattering) lighting components of a scene from novel viewpoints, using a small number of multi-view images. By integrating neural radiance fields (NeRF) with a projector-camera system, they achieve robust separation even with image blur, enabling applications like material editing and improved 3D shape recovery.

What Problem Does This Paper Address?

Understanding how light interacts with a scene is crucial for many computer vision and graphics tasks, such as realistic rendering, material editing, and 3D reconstruction. A scene's observed radiance is a combination of "direct" light (coming straight from the light source) and "global" light (light that has bounced around the scene). Separating these components is challenging, especially when dealing with complex lighting phenomena like inter-reflections or subsurface scattering.

Existing methods often struggle with several limitations:

  • Viewpoint Dependency: They typically require images captured from the same viewpoint to perform separation, meaning you need many images for each desired viewpoint.
  • Image Requirements: Many methods need a large number of images (e.g., 25) to achieve reasonable results, often due to artifacts caused by projector blur and limited camera resolution.
  • Robustness to Blur: Projectors, especially consumer-grade ones, have shallow depth of field, leading to blurred patterns. This blur can cause artifacts in the separated components.

This paper aims to overcome these limitations by enabling direct-global separation from novel viewpoints using a small number of images, while also being robust to projector blur.

Methods

The core of the proposed method lies in combining two key technologies:

  1. Projector-Camera System: The authors use a coaxial projector-camera system, meaning the projector and camera share the same optical axis. This is achieved using a half-mirror, which simplifies the correspondence between projector pixels and camera pixels, making it invariant to scene depth. They project a black-and-white checkered pattern onto the scene.
  2. Neural Radiance Fields (NeRF): Instead of traditional image processing techniques, they leverage NeRF's ability to represent a scene as a continuous function. Their NeRF network takes a 5D coordinate (x, y, z, viewing direction) as input and outputs not only the scene's density and color but also the density and radiance values for both the direct and global components separately.

Key methodological innovations:

  • NeRF for Separation: The network is designed to predict direct and global components directly, rather than relying on post-processing of a single output.
  • Observation Model: They use a modified observation model where the observed pixel value $I_v(i, j)$ is a combination of the predicted direct component $D_v(i, j)$ and global component $G_v(i, j)$, modulated by the illumination condition $L_v(i, j)$ (1 for white light, 0 for black light in the checkered pattern): $I_v(i, j) = L_v(i, j) D_v(i, j) + L_v G_v(i, j)$.
  • Weighted Loss Function: To combat blur artifacts near the boundaries of the checkered pattern, they introduce a weighted loss function. This weight $W(i, j)$ is set to 0 for pixels near boundaries and 1 otherwise, gradually reducing the influence of blurry regions as training progresses. This allows the network to focus on more reliable pixels over time.
  • Novel View Synthesis: Once the NeRF is trained, it can synthesize direct and global components from any novel viewpoint through volume rendering.

Datasets

The paper uses real-world captured data. They do not introduce a new benchmark dataset but rather collect image sequences from various scenes using their projector-camera setup.

  • Scenes: The experiments were conducted on scenes including ping-pong balls, plastic toys, candles, fruits, and flowers.
  • Data Capture: They captured image sequences by continuously moving the scenes in front of the projector-camera system. For training, they used a varying number of images per scene (e.g., 180 images for some experiments, 128 for others). For evaluation, they used novel viewpoints not seen during training.

Key Insights and Ideas

  • NeRF's Potential Beyond View Synthesis: This work demonstrates that NeRF architectures can be extended beyond just synthesizing novel views to perform complex scene decomposition tasks like direct-global separation.
  • Leveraging Projector-Camera for NeRF: The coaxial projector-camera setup is crucial. It provides known illumination conditions and simplifies pixel correspondence, which are essential for training the NeRF to disentangle components.
  • Blur as a Feature (with caveats): While blur is often an artifact, the paper shows that by carefully weighting the loss function, the network can learn to mitigate its negative impact and still extract meaningful components. The "varying weight" strategy is key here.
  • Direct/Global Invariance: The authors exploit the observation that direct and global components are generally less sensitive to viewpoint changes than the combined image. This allows for interpolation and synthesis from novel views.

Key Results

  • Qualitative Success: The method successfully synthesizes direct and global components from novel viewpoints, capturing subtle effects like inter-reflections and subsurface scattering in the global component.
  • Quantitative Performance: In comparisons against methods that use two images per view (Method A) or a single image with a stripe pattern (Method B), their method (using a single checkered pattern image per view) consistently achieved higher PSNR and SSIM scores across multiple scenes. For example, for "balls," their method achieved PSNRs of 36.62 (direct) and 41.72 (global), significantly outperforming Method A (24.54/24.79) and Method B (23.29/24.45).
  • Robustness to Blur: The ablation study showed that their "varying weight" training strategy significantly reduced grid-like artifacts compared to a "static weight" approach, leading to better quantitative results.
  • Applications: The method demonstrated utility in:
  • Image-based Material Editing: Recombining direct and global components with different weights allowed for enhanced specular reflections or richer inter-reflections.
  • 3D Shape Recovery: Synthesizing direct components from novel views improved the accuracy of 3D shape recovery, particularly for fine details like dents that were lost in the global component.

Limitations and Future Work

  • Coaxial System Assumption: The current method relies on a coaxial projector-camera system, which implies known illumination conditions.
  • Future Work:
    • Arbitrary Viewpoints/Light Sources: Extending the method to handle arbitrary viewpoints and independent light source positions is a key direction.
  • Improved Projection Patterns: While the checkered pattern works well, exploring other projection patterns could further enhance performance.

Summary updated

BibTeX
@inproceedings{matsufuji2025separating,
  author = {Kengo Matsufuji and Lin Shi and Ryo Kawahara and Takahiro Okabe},
  title = {{Separating Direct and Global Components from Novel Viewpoints}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure