TL;DR
This paper introduces a novel approach to photometric stereo using neural networks and "illumination planning" to overcome challenges posed by cast shadows. By intelligently selecting which light sources to use for each point on an object, the method significantly improves the accuracy of 3D shape recovery, especially in regions previously obscured by shadows.
What Problem Does This Paper Address?
Photometric stereo is a technique for reconstructing the 3D shape of an object by analyzing how its surface appearance changes under different lighting conditions. The core idea is that the brightness of a surface point depends on its orientation (surface normal) and the direction of the light source. However, this technique faces significant challenges in practice. One major issue is the presence of shadows, particularly "cast shadows," where one part of the object blocks light from reaching another. In these shadowed regions, the standard photometric stereo equations break down, leading to inaccurate or missing shape information. Furthermore, when light sources are nearly collinear, the problem becomes numerically ill-conditioned, making accurate estimation difficult. Existing methods often struggle to robustly handle these issues, especially with complex object shapes.
Methods
The authors propose a two-pronged approach combining neural networks with a new "illumination planning" strategy.
- Neural Network Architecture: They employ Radial Basis Function (RBF) neural networks for non-parametric functional approximation. Two RBF networks are trained: one maps image irradiance to surface normal, and a second, inverse network maps surface normal back to image irradiance. This inverse mapping is crucial for estimating confidence.
- Principal Components Analysis (PCA): To handle the ill-conditioning caused by nearly collinear light sources and reduce dimensionality, PCA is used as a preprocessing step. It transforms the image irradiance data into a lower-dimensional subspace, making it more amenable for the neural networks.
- Illumination Planning: This is the core novel contribution. The process involves:
- Initial Recovery: Using all available light sources, an initial shape estimate and a "resynthesized" input are generated by the trained neural networks.
- Confidence Estimation: A confidence measure for each pixel is calculated by comparing the original input data with the resynthesized input. This difference highlights regions where the neural network struggled, often due to shadows.
- Cast Shadow Identification: Based on the confidence measure, regions affected by cast shadows are identified. The method specifically targets "cast shadows" where the irradiance values are unreliable.
- Selective Recovery: For the identified cast shadow regions, the illumination planning strategy intelligently disposes of ill-effected light sources. The system then re-recovers the shape using only the well-effected light sources for those specific points.
Datasets
The paper uses a pottery sphere for calibration. This choice is motivated by its convex nature (minimizing interreflections), its ability to span all visible surface normals, and its ease of geometric analysis. A pottery moon face is used as the test object for evaluation. The experiments do not rely on specific assumptions about reflectance properties or light source directions, as long as the calibration and test objects are viewed under the same illumination pattern and are made of the same material.
Key Insights and Ideas
- Confidence as a Shadow Detector: The ingenious use of an inverse neural network to predict the input from the output provides a robust confidence measure. Discrepancies between the actual and predicted inputs directly indicate regions where the photometric stereo model is likely failing, often due to cast shadows.
- Intelligent Light Source Selection: Instead of treating all light sources equally, the illumination planning dynamically selects the most informative subset of light sources for each specific point on the object. This is a significant departure from traditional methods that use all available lights.
- PCA for Ill-Conditioning: Applying PCA to the image data effectively addresses the numerical instability that arises when light source directions are too similar, making the neural network training more stable and the results more reliable.
- Non-Parametric Robustness: The approach is empirical and non-parametric, meaning it doesn't require prior knowledge of exact light source directions or complex surface reflectance models, making it more broadly applicable.
Key Results
The paper demonstrates significant improvements in shape recovery, particularly in regions affected by cast shadows.
- Quantitative Improvement: The mean confidence estimate,
mean(C), improved from 4.73629 before illumination planning to 4.11619 after. This represents a numerical improvement of 13.1%. - Qualitative Improvement:
- "Needle maps" (showing surface normals) and "slope maps" show a reduction in disconnected or erroneous vectors in areas like the eyes, mouth, and arms of the moon face after illumination planning.
- Confidence maps visually confirm that regions previously showing low confidence (bright areas) are improved, indicating more reliable shape estimates.
- The method successfully recovers shape in regions previously affected by cast shadows, in addition to the normal reflection regions.
Limitations and Future Work
- Limitations: The paper acknowledges that "interreflection regions" can still affect the results, although they are less problematic than cast shadows. The current method focuses on cast shadows.
- Future Work: The authors suggest future research directions including:
- Applying the NN approach to non-uniform test objects.
- Developing methods for correcting interreflections.
Summary updated