Introduction
In computational photography, distinguishing between light that reflects directly from a surface and light that undergoes global transport—such as subsurface scattering or inter-reflections—is essential for high-fidelity 3D shape recovery and material analysis. Traditional direct-global separation techniques rely on projecting high-frequency patterns onto a scene. However, these methods typically require the scene and camera to remain perfectly static across multiple frames to prevent motion artifacts.
When objects move, the changing illumination patterns break the fundamental assumptions of standard optical flow algorithms, resulting in severe "ghosting" in the separated components. This research introduces a robust pipeline to compensate for motion, enabling high-resolution separation in dynamic video sequences.
Methodology
The proposed approach addresses the challenges of dynamic scenes through a three-step computational pipeline:
- Relighting Approximation: The system first estimates a "relit" version of each frame—an approximation of the scene under uniform illumination. This step effectively strips away the projected patterns to create a stable reference for tracking.
- Robust Registration: Using these relit images, the method computes optical flow to track pixel movement. This alignment is refined with a specialized cost function that utilizes both the relit estimates and the original patterned data to achieve sub-pixel accuracy.
- Joint Separation and Regularization: With aligned frames, the direct and global components are solved for simultaneously. The process employs Total Variation (TV) regularization, leveraging the physical reality that global light transport is spatially smooth while direct reflection contains high-frequency details.
Key Insights
A critical insight of this work is that perfect separation is not required for successful alignment; a "good enough" relit estimate allows standard registration algorithms to function. By utilizing every frame for both tracking and separation, the method is significantly more efficient than previous interleaving techniques that sacrifice frames for tracking purposes. Furthermore, the use of a beamsplitter to co-locate the projector and camera simplifies the geometry, avoiding the need for complex 3D reconstruction during alignment.
Results and Discussion
The algorithm was validated on rigid objects and deformable scenes, such as human faces and moving hands. Experimental results demonstrate that the "Moving Refined" algorithm achieves accuracy levels nearly identical to static ground-truth captures. Unlike single-shot methods that trade spatial resolution for speed, this approach maintains the full resolution of the camera, successfully isolating fine surface details in the direct component and the soft glow of materials in the global component.
Conclusion
By integrating motion compensation into the separation process, this work enables high-quality material analysis and 3D scanning for dynamic real-world environments. Future research may explore handling specular reflections and extremely rapid motion through advanced deblurring or coded aperture techniques.
Summary updated