Neural Network Based Photometric Strereo Using Illumination Planning

Yuji Iwahori, Wataru Kato, Shoaib Bhuiyab, Robert J. Woodham, Naohiro IshiiIJCAI 1997

Abstract

Photometric stereo is a powerful technique for recovering 3D surface geometry from images under varying illumination. However, traditional physics-based models often struggle with complex materials and cast shadows. This paper introduces a robust framework using dual Radial Basis Function (RBF) neural networks combined with Principal Component Analysis (PCA). A novel "illumination planning" strategy is employed, utilizing an inverse network to detect and mitigate the effects of cast shadows. This self-checking mechanism allows the system to identify corrupted data and re-calculate surface orientations using only reliable light sources. Experimental results on a complex "Moon Face" object demonstrate a 13.1% improvement in reconstruction confidence, highlighting the system's efficacy in handling real-world geometric challenges.

Problem Statement

The recovery of surface normals through photometric stereo faces two primary challenges:

  1. Numerical Instability (Ill-Conditioning): When light sources are positioned close to one another or the viewing direction, the resulting equations become sensitive to noise, leading to significant errors in shape estimation.
  2. Cast Shadows: Standard reflectance models assume every surface point is visible to every light source. In complex geometries, features like a nose or brow often cast shadows on other parts of the object, rendering the irradiance data for those pixels misleading.

Proposed Methodology

The researchers move away from rigid theoretical models toward an empirical, learning-based pipeline. The process involves three key stages:

Dimensionality Reduction

To address the instability of nearby light sources, Principal Component Analysis (PCA) is used to compress irradiance data from multiple sources into a stable 3D subspace. This filtering process removes redundancy and noise, providing a more robust input for the neural networks.

Dual RBF Neural Networks

The core of the system consists of two interconnected Radial Basis Function networks:

  • Forward Network (NN1): Maps the compressed image irradiance values directly to the estimated surface normal.
  • Inverse Network (NN2): Maps the estimated surface normal back to the expected image irradiance.
The proposed dual RBF neural network pipeline incorporating PCA for dimensionality reduction and illumination planning.

Illumination Planning

This "self-checking" mechanism is the system's primary innovation. By comparing the actual observed brightness of a pixel to the brightness predicted by the inverse network (NN2), the system calculates a confidence estimate. A high discrepancy indicates a cast shadow. In such cases, the system identifies the problematic light source, discards its data, and re-estimates the surface orientation using the remaining valid sources.

Experimental Setup and Datasets

The study utilized two objects made of the same pottery material:

  1. Pottery Sphere: Used for calibration. The known geometric properties of the sphere allow the networks to learn the specific reflectance characteristics of the material.
  2. Pottery "Moon Face": A test object with complex features (deep-set eyes and mouth) designed to test the system's ability to handle significant cast shadows.

Results and Performance

The implementation of illumination planning yielded significant improvements in both quantitative and qualitative metrics:

  • Shadow Mitigation: The system successfully identified and corrected for cast shadows in the recessed areas of the Moon Face.
  • Accuracy Gains: The mean confidence error was reduced by 13.1% following the application of the illumination planning strategy.
  • Visual Consistency: Before planning, surface visualizations (needle maps) appeared noisy and disconnected in shadowed regions. Post-planning, the reconstructed surfaces became smooth and consistent, accurately capturing the object's contours.
Reconstruction of the 'Moon Face' test object, demonstrating the system's ability to handle complex geometries and shadows.

Key Insights

  • Diagnostic Inverse Networks: Using an inverse network as a diagnostic tool allows the system to validate its own predictions against physical reality.
  • Implicit Physics: By training on a calibration object, the RBF networks implicitly learn complex material properties without requiring explicit mathematical modeling of non-Lambertian reflectance.
  • Subspace Stability: PCA proves that increasing the number of light sources is most effective when the data is projected into a stable coordinate system, reducing sensitivity to physical lamp placement.

Limitations and Future Directions

While effective, the current model has limitations. It requires the test object to be made of the same uniform material as the calibration object, preventing its use on multi-material surfaces. Additionally, errors introduced by interreflections (light bouncing between surface parts) are not yet fully addressed. Future work will focus on adapting the system for arbitrary lighting patterns and non-uniform materials.

Summary updated

BibTeX
@inproceedings{iwahori1997neural,
  author = {Yuji Iwahori and Wataru Kato and Shoaib Bhuiyab and Robert J. Woodham and Naohiro Ishii},
  title = {{Neural Network Based Photometric Strereo Using Illumination Planning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1997},
}
Figure