Multiple Light Sources Recovery in the Presence of Specularities and Texture

Pascal Lagger, Pascal FuaInfoscience 2005

Abstract

Recovering light sources from images of moving objects is difficult because specularities travel along surfaces and diffuse reflections change. This paper presents a method to recover the number, direction, and intensity of multiple light sources by leveraging 3D models. The approach involves tracking local gray-level maxima to distinguish specularities from texture. These specularities are then used to estimate light source locations and material properties. The algorithm is robust to arbitrary textures and multiple light sources of unknown colors.

Introduction: The Inverse Rendering Challenge

In computer vision, "inverse rendering"—the process of estimating lighting and material properties from images—presents a classic "chicken and egg" problem. Determining the light requires knowledge of the material, and vice versa. Traditional methods often simplify the problem by assuming Lambertian (perfectly matte) surfaces or uniform colors. However, real-world objects are frequently shiny and highly textured. Specular highlights are particularly challenging because they move across a surface as the object or camera shifts, often being confused with bright texture elements. This research addresses these challenges by treating specularities as valuable information rather than noise.

Methodology: Tracking Specularities

The proposed pipeline requires a 3D model of the object, which can be a rough approximation. It consists of three primary stages:

  1. Mean-Shift Maxima Detection: Instead of simple intensity thresholding, which fails when texture is brighter than highlights, a mean-shift algorithm treats the image as a density distribution. This identifies local brightness "peaks," allowing the system to detect even weak specularities.
Detection of local intensity maxima (yellow) used to distinguish specular highlights from surface texture.
  1. Consistency Voting: To distinguish true specularities from bright texture, the algorithm projects detected peaks onto the 3D mesh. If a peak remains at the same mesh coordinate during motion, it is labeled as texture. If it moves in a manner consistent with reflection geometry, it is identified as a specularity. These specularities then "vote" for light source directions in a 2D accumulator.
  2. Optimization and Refinement: Once initial light directions are found, parameters are refined via non-linear optimization. This step minimizes the difference between the actual images and a rendered model, solving for light intensities and the object's unlighted texture map (albedo).

Experimental Results

The algorithm was evaluated using real-world and synthetic sequences, including a teapot, an industrially painted Easter egg, and a human face.

The recovery process for the teapot sequence: (a, b) input images, (c) diffuse component, and (d) the final recovered texture map.

In the teapot experiment, the system successfully identified three distinct light sources of different colors. For the Easter egg, the algorithm decoupled lighting from texture, producing a clean albedo map that removed all highlights and shading. Notably, the method proved robust to geometric inaccuracies; even a generic face model fitted to images was sufficient to recover lighting parameters, demonstrating practical utility for real-world scenarios where high-resolution scans are unavailable.

Conclusion

By treating specularities as "sensors" that reflect the lighting environment, this method provides a robust solution for inverse rendering on textured, moving objects. While the current model assumes distant light sources and does not explicitly account for shadows, it successfully demonstrates that motion and specularities can be leveraged to solve complex lighting recovery problems. Future work will focus on incorporating spatio-temporal constraints and shadow modeling to further enhance material property estimation.

Summary updated

Abstract

Recovering the light sources from images in which the objects can move is difficult both because the specularities travel along the surfaces and because diffuse reflections do not remain constant. In this paper, we show that this can nevertheless be done without making the very restrictive assumptions that are usually made provided that 3D models of some scene objects are available or can be computed. Our approach involves following local gray-level maxima and deciding whether or not they represent specularities. The true specularities are then used to count how many light sources are in the scene, compute their precise location and estimate the material properties. The resulting algorithm can operate in presence of arbitrary textures and multiple light sources of possibly different unknown colors. 1.
BibTeX
@inproceedings{lagger2005multiple,
  author = {Pascal Lagger and Pascal Fua},
  title = {{Multiple Light Sources Recovery in the Presence of Specularities and Texture}},
  booktitle = {Infoscience},
  year = {2005},
}
Figure