EvDiG: Event-guided Direct and Global Components Separation

Xinyu Zhou, Peiqi Duan, Boyu Li, Chu Zhou, Chao Xu, Boxin Shi

Abstract

Separating direct and global illumination components is a fundamental task in computer vision, yet existing methods face a stark trade-off between acquisition speed and reconstruction quality. Multi-frame methods provide high fidelity but are slow, while single-frame methods are fast but often inaccurate. We present EvDiG, a novel hybrid camera system that leverages an event camera and a sweeping shadow to achieve high-fidelity separation at video rates (33ms). By capturing rapid radiance changes with microsecond precision, EvDiG reconstructs illumination components with 20x the efficiency of state-of-the-art multi-frame approaches while outperforming single-frame alternatives.

Introduction

When light interacts with a scene, it follows two primary paths: direct illumination (light that hits a surface once and reflects to the camera) and global illumination (light that bounces multiple times, scatters, or diffuses). Separating these components is critical for downstream tasks like 3D shape recovery and material analysis.

Traditional multi-frame methods are accurate but require capturing dozens of images under different lighting patterns, making them unsuitable for dynamic scenes. Conversely, single-frame methods are fast but often produce blurry results as they rely on priors to "guess" missing information. EvDiG bridges this gap by using a hybrid system consisting of a standard RGB camera, an event camera, and a simple "stick" occluder that sweeps across the scene.

Overview of the EvDiG system, input types, and performance comparison.

Methodology

The core innovation of EvDiG is the use of an event camera to track a moving shadow. Unlike standard cameras, event cameras only report changes in brightness with microsecond precision, making them ideal for capturing the exact moment a pixel enters or leaves a shadow.

Event-Based Reconstruction

As the occluder's shadow sweeps across the scene, the system records radiance changes. By accumulating these events, the system reconstructs the minimum and maximum brightness for every pixel. This provides a coarse initial separation of direct and global light.

Refinement Networks: EvSepNet and ImColorNet

To handle noise and restore color, the authors designed a two-stage neural architecture:

  1. EvSepNet: A U-Net-style network that includes a Separation Correction Block (SCB). It uses spatial attention to reweight features, effectively cleaning up artifacts from high-frequency global illumination in the grayscale domain.
  2. ImColorNet: Since event data is typically grayscale, this network restores color using a Color Correction Block (CCB). Inspired by self-attention mechanisms, it intelligently maps color from the original RGB frame onto the separated components, accounting for the fact that global light's spectral distribution changes as it bounces off colored surfaces.
The EvDiG pipeline featuring EvSepNet and ImColorNet architectures.

Experimental Results

The authors evaluated EvDiG using a new dataset of 230 indoor scenes and 1800 clips. They used high-quality multi-frame results as pseudo-ground truth for training.

  • Efficiency: EvDiG requires only 33ms of data collection (one inter-frame period at 30 fps), representing a 20x speedup over traditional multi-frame methods (833ms - 1000ms).
  • Accuracy: Quantitatively, EvDiG achieved a PSNR of 30.01 for direct components and 31.66 for global components, significantly outperforming deep-learning-based single-frame methods.
  • Robustness: The system was also successfully tested on complex outdoor scenes, proving its real-world applicability.

Conclusion and Future Work

EvDiG demonstrates that event cameras are the "missing link" for efficient light component separation, providing high temporal resolution without massive computational overhead. While the current method relies on a physical occluder, future research could explore using natural shadows or integrating these hybrid sensors into mobile devices. Despite its speed, the system still assumes a relatively static scene during the short shadow-pass window, a limitation that future iterations may address.

Reference: Zhou, X., et al. "EvDiG: Event-guided Direct and Global Components Separation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

Summary updated

Abstract

Separating the direct and global components of a scene aids in shape recovery and basic material understanding. Conventional methods capture multiple frames under high frequency illumination patterns or shadows, requiring the scene to keep stationary during the image acquisition process. Single-frame methods simplify the capture procedure but yield lower-quality separation results. In this paper, we leverage the event camera to facilitate the separation of direct and global components, enabling video-rate separation of high quality. In detail, we adopt an event camera to record rapid illumination changes caused by the shadow of a line occluder sweeping over the scene, and reconstruct the coarse separation results through event accumulation. We then design a network to resolve the noise in the coarse sep-aration results and restore color information. A real-world dataset is collected using a hybrid camera system for network training and evaluation. Experimental results show superior performance over state-of-the-art methods.
BibTeX
@inproceedings{zhou2024evdig,
  author = {Xinyu Zhou and Peiqi Duan and Boyu Li and Chu Zhou and Chao Xu and Boxin Shi},
  title = {{EvDiG: Event-guided Direct and Global Components Separation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00918},
}
Figure