Development of Real Time System for Synthesizing Relighting Image by using RGB-D Camera

Takuya Ikeda, Yohei Ogura, François de Sorbier, Hideo SaitôThe Journal of The Institute of Image Information and Television Engineers 2014 · DOI

Abstract

Image relighting involves synthesizing an image of a subject as if it were captured in a different lighting environment. Achieving this in real-time requires precise 3D geometry (surface normals), reflection characteristics, and lighting environment data. This paper proposes a system capable of real-time relighting using a consumer-grade RGB-D camera. To overcome the inherent noise in depth data, we implement a depth map refinement process for accurate object extraction and surface normal estimation. Lighting is represented via Spherical Harmonics derived from a mirrored sphere, and the entire pipeline is GPU-accelerated. Our implementation achieves a processing rate of approximately 10 frames per second (fps).

Introduction

The ability to change the illumination of a subject in a video stream to match a target environment—known as image relighting—has significant applications in augmented reality and virtual production. Traditionally, high-fidelity relighting required specialized hardware like "Light Stages" or pre-computed, high-resolution 3D models.

Implementing this using standard RGB-D sensors (e.g., Microsoft Kinect) presents three primary challenges:

  1. Sensor Noise: Depth data is often noisy, particularly at object boundaries, leading to visual artifacts.
  2. Normal Estimation: Accurate surface normals are essential for calculating light interaction, but computing them from noisy depth data in real-time is computationally expensive.
  3. Latency: The mathematical complexity of integrating environmental light requires significant optimization to maintain interactive frame rates.

Methodology

The proposed system utilizes a multi-stage pipeline optimized for the GPU to ensure real-time performance.

Depth Map Refinement and Object Extraction

To address the "halo" effect caused by misalignment between color and depth sensors, we employ a combination of Bilateral Filtering and k-means clustering. This refinement ensures that the object mask derived from the depth map aligns precisely with the high-resolution color image.

The mannequin subject used for system evaluation and quantitative testing.

Surface Normal Estimation

We propose a "CM + Bilateral" approach for normal estimation. This method leverages Integral Images to rapidly calculate surface curvature (CM), followed by a bilateral filter to smooth the resulting normals without degrading sharp geometric features. This approach provides a superior balance between computational speed and visual quality compared to standard kNN or raw depth-based methods.

Lighting and Shading

The system assumes Lambertian (matte) surfaces to simplify the relighting equation. By representing the target environment's lighting using Spherical Harmonics (SH) coefficients, we can perform complex lighting calculations extremely efficiently. The final shading is calculated as a ratio of irradiance between the source and destination environments.

Experimental Results

The system was evaluated using both static objects and dynamic human subjects.

Performance and Accuracy

The GPU-accelerated pipeline achieved a stable throughput of 10 fps at 640x480 resolution. Quantitative tests using a mannequin head as a ground truth demonstrated that our refined normal estimation significantly reduced Root Mean Square Error (RMSE) compared to using unrefined depth data.

A relit image generated by the system, showing realistic highlights and shadows that match the target environment.

Visual Quality

Qualitative results indicate that the system successfully synthesizes realistic lighting effects. For instance, subjects placed in a dark virtual environment and lit by a simulated side-lamp showed appropriate highlights and shadowing consistent with the light source's position.

Discussion and Future Work

While the system demonstrates the feasibility of real-time RGB-D relighting, several limitations remain. The current Lambertian assumption precludes the accurate rendering of specular (shiny) materials. Furthermore, the system does not yet account for complex self-shadowing (e.g., shadows cast by a subject's chin). Future iterations will explore Screen Space Ambient Occlusion (SSAO) to improve shadow realism and dynamic environment capture to remove the need for pre-captured mirror ball data.

Conclusion

We have developed a real-time relighting system that leverages consumer RGB-D hardware and GPU acceleration. By focusing on depth refinement and efficient normal estimation, the system provides a practical solution for interactive relighting at 10 fps, paving the way for more immersive AR and VR experiences.

Summary updated

Abstract

リライティングとは,実画像から,それを撮影した照明環境を別の正面環境で撮影したかのような画像を合成する処理のことであり,その実現には,対象物体の3次元形状(法線分布),反射特性,光源環境が必要である.本論文では,RGB-Dカメラを用いて実時間リライティングを実現可能なシステムを提案する.本システムでは,正確な物体領域抽出のためにノイズを含む距離画像の修正を行う.領域抽出を行った後,対象物体の法線分布を推定する.光源環境は,観測された鏡面球から取得し,任意の環境に対してリライティングを行う.また,実時間処理を実現するために,GPUを用いた並列処理を行う.我々の実装では,約10fpsの速度でリライティングが可能である.提案システムの有効性を確認するために行った実験では,対象物体に対して事前に撮影した動的光源環境へリライティングを行い,提案システムの性能を評価した.
BibTeX
@article{ikeda2014development,
  author = {Takuya Ikeda and Yohei Ogura and François de Sorbier and Hideo Saitô},
  title = {{Development of Real Time System for Synthesizing Relighting Image by using RGB-D Camera}},
  journal = {The Journal of The Institute of Image Information and Television Engineers},
  year = {2014},
  doi = {10.3169/itej.68.j558},
}
Figure