Abstract
Separating direct and global illumination components is a fundamental task in computer vision, yet existing methods face a stark trade-off between acquisition speed and reconstruction quality. Multi-frame methods provide high fidelity but are slow, while single-frame methods are fast but often inaccurate. We present EvDiG, a novel hybrid camera system that leverages an event camera and a sweeping shadow to achieve high-fidelity separation at video rates (33ms). By capturing rapid radiance changes with microsecond precision, EvDiG reconstructs illumination components with 20x the efficiency of state-of-the-art multi-frame approaches while outperforming single-frame alternatives.
Introduction
When light interacts with a scene, it follows two primary paths: direct illumination (light that hits a surface once and reflects to the camera) and global illumination (light that bounces multiple times, scatters, or diffuses). Separating these components is critical for downstream tasks like 3D shape recovery and material analysis.
Traditional multi-frame methods are accurate but require capturing dozens of images under different lighting patterns, making them unsuitable for dynamic scenes. Conversely, single-frame methods are fast but often produce blurry results as they rely on priors to "guess" missing information. EvDiG bridges this gap by using a hybrid system consisting of a standard RGB camera, an event camera, and a simple "stick" occluder that sweeps across the scene.
Methodology
The core innovation of EvDiG is the use of an event camera to track a moving shadow. Unlike standard cameras, event cameras only report changes in brightness with microsecond precision, making them ideal for capturing the exact moment a pixel enters or leaves a shadow.
Event-Based Reconstruction
As the occluder's shadow sweeps across the scene, the system records radiance changes. By accumulating these events, the system reconstructs the minimum and maximum brightness for every pixel. This provides a coarse initial separation of direct and global light.
Refinement Networks: EvSepNet and ImColorNet
To handle noise and restore color, the authors designed a two-stage neural architecture:
- EvSepNet: A U-Net-style network that includes a Separation Correction Block (SCB). It uses spatial attention to reweight features, effectively cleaning up artifacts from high-frequency global illumination in the grayscale domain.
- ImColorNet: Since event data is typically grayscale, this network restores color using a Color Correction Block (CCB). Inspired by self-attention mechanisms, it intelligently maps color from the original RGB frame onto the separated components, accounting for the fact that global light's spectral distribution changes as it bounces off colored surfaces.
Experimental Results
The authors evaluated EvDiG using a new dataset of 230 indoor scenes and 1800 clips. They used high-quality multi-frame results as pseudo-ground truth for training.
- Efficiency: EvDiG requires only 33ms of data collection (one inter-frame period at 30 fps), representing a 20x speedup over traditional multi-frame methods (833ms - 1000ms).
- Accuracy: Quantitatively, EvDiG achieved a PSNR of 30.01 for direct components and 31.66 for global components, significantly outperforming deep-learning-based single-frame methods.
- Robustness: The system was also successfully tested on complex outdoor scenes, proving its real-world applicability.
Conclusion and Future Work
EvDiG demonstrates that event cameras are the "missing link" for efficient light component separation, providing high temporal resolution without massive computational overhead. While the current method relies on a physical occluder, future research could explore using natural shadows or integrating these hybrid sensors into mobile devices. Despite its speed, the system still assumes a relatively static scene during the short shadow-pass window, a limitation that future iterations may address.
Reference: Zhou, X., et al. "EvDiG: Event-guided Direct and Global Components Separation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Summary updated