Abstract
Image relighting involves synthesizing an image of a subject as if it were captured in a different lighting environment. Achieving this in real-time requires precise 3D geometry (surface normals), reflection characteristics, and lighting environment data. This paper proposes a system capable of real-time relighting using a consumer-grade RGB-D camera. To overcome the inherent noise in depth data, we implement a depth map refinement process for accurate object extraction and surface normal estimation. Lighting is represented via Spherical Harmonics derived from a mirrored sphere, and the entire pipeline is GPU-accelerated. Our implementation achieves a processing rate of approximately 10 frames per second (fps).
Introduction
The ability to change the illumination of a subject in a video stream to match a target environment—known as image relighting—has significant applications in augmented reality and virtual production. Traditionally, high-fidelity relighting required specialized hardware like "Light Stages" or pre-computed, high-resolution 3D models.
Implementing this using standard RGB-D sensors (e.g., Microsoft Kinect) presents three primary challenges:
- Sensor Noise: Depth data is often noisy, particularly at object boundaries, leading to visual artifacts.
- Normal Estimation: Accurate surface normals are essential for calculating light interaction, but computing them from noisy depth data in real-time is computationally expensive.
- Latency: The mathematical complexity of integrating environmental light requires significant optimization to maintain interactive frame rates.
Methodology
The proposed system utilizes a multi-stage pipeline optimized for the GPU to ensure real-time performance.
Depth Map Refinement and Object Extraction
To address the "halo" effect caused by misalignment between color and depth sensors, we employ a combination of Bilateral Filtering and k-means clustering. This refinement ensures that the object mask derived from the depth map aligns precisely with the high-resolution color image.
Surface Normal Estimation
We propose a "CM + Bilateral" approach for normal estimation. This method leverages Integral Images to rapidly calculate surface curvature (CM), followed by a bilateral filter to smooth the resulting normals without degrading sharp geometric features. This approach provides a superior balance between computational speed and visual quality compared to standard kNN or raw depth-based methods.
Lighting and Shading
The system assumes Lambertian (matte) surfaces to simplify the relighting equation. By representing the target environment's lighting using Spherical Harmonics (SH) coefficients, we can perform complex lighting calculations extremely efficiently. The final shading is calculated as a ratio of irradiance between the source and destination environments.
Experimental Results
The system was evaluated using both static objects and dynamic human subjects.
Performance and Accuracy
The GPU-accelerated pipeline achieved a stable throughput of 10 fps at 640x480 resolution. Quantitative tests using a mannequin head as a ground truth demonstrated that our refined normal estimation significantly reduced Root Mean Square Error (RMSE) compared to using unrefined depth data.
Visual Quality
Qualitative results indicate that the system successfully synthesizes realistic lighting effects. For instance, subjects placed in a dark virtual environment and lit by a simulated side-lamp showed appropriate highlights and shadowing consistent with the light source's position.
Discussion and Future Work
While the system demonstrates the feasibility of real-time RGB-D relighting, several limitations remain. The current Lambertian assumption precludes the accurate rendering of specular (shiny) materials. Furthermore, the system does not yet account for complex self-shadowing (e.g., shadows cast by a subject's chin). Future iterations will explore Screen Space Ambient Occlusion (SSAO) to improve shadow realism and dynamic environment capture to remove the need for pre-captured mirror ball data.
Conclusion
We have developed a real-time relighting system that leverages consumer RGB-D hardware and GPU acceleration. By focusing on depth refinement and efficient normal estimation, the system provides a practical solution for interactive relighting at 10 fps, paving the way for more immersive AR and VR experiences.
Summary updated