Abstract
Recovering light sources from images of moving objects is difficult because specularities travel along surfaces and diffuse reflections change. This paper presents a method to recover the number, direction, and intensity of multiple light sources by leveraging 3D models. The approach involves tracking local gray-level maxima to distinguish specularities from texture. These specularities are then used to estimate light source locations and material properties. The algorithm is robust to arbitrary textures and multiple light sources of unknown colors.
Introduction: The Inverse Rendering Challenge
In computer vision, "inverse rendering"—the process of estimating lighting and material properties from images—presents a classic "chicken and egg" problem. Determining the light requires knowledge of the material, and vice versa. Traditional methods often simplify the problem by assuming Lambertian (perfectly matte) surfaces or uniform colors. However, real-world objects are frequently shiny and highly textured. Specular highlights are particularly challenging because they move across a surface as the object or camera shifts, often being confused with bright texture elements. This research addresses these challenges by treating specularities as valuable information rather than noise.
Methodology: Tracking Specularities
The proposed pipeline requires a 3D model of the object, which can be a rough approximation. It consists of three primary stages:
- Mean-Shift Maxima Detection: Instead of simple intensity thresholding, which fails when texture is brighter than highlights, a mean-shift algorithm treats the image as a density distribution. This identifies local brightness "peaks," allowing the system to detect even weak specularities.
- Consistency Voting: To distinguish true specularities from bright texture, the algorithm projects detected peaks onto the 3D mesh. If a peak remains at the same mesh coordinate during motion, it is labeled as texture. If it moves in a manner consistent with reflection geometry, it is identified as a specularity. These specularities then "vote" for light source directions in a 2D accumulator.
- Optimization and Refinement: Once initial light directions are found, parameters are refined via non-linear optimization. This step minimizes the difference between the actual images and a rendered model, solving for light intensities and the object's unlighted texture map (albedo).
Experimental Results
The algorithm was evaluated using real-world and synthetic sequences, including a teapot, an industrially painted Easter egg, and a human face.
In the teapot experiment, the system successfully identified three distinct light sources of different colors. For the Easter egg, the algorithm decoupled lighting from texture, producing a clean albedo map that removed all highlights and shading. Notably, the method proved robust to geometric inaccuracies; even a generic face model fitted to images was sufficient to recover lighting parameters, demonstrating practical utility for real-world scenarios where high-resolution scans are unavailable.
Conclusion
By treating specularities as "sensors" that reflect the lighting environment, this method provides a robust solution for inverse rendering on textured, moving objects. While the current model assumes distant light sources and does not explicitly account for shadows, it successfully demonstrates that motion and specularities can be leveraged to solve complex lighting recovery problems. Future work will focus on incorporating spatio-temporal constraints and shadow modeling to further enhance material property estimation.
Summary updated