LIPIDS: Learning-Based Illumination Planning in Discretized (Light) Space for Photometric Stereo

Ashish Tiwari, Mihirkumar Sutariya, Shanmuganathan RamanWACV 2025

TL;DR

This paper introduces LIPIDS, a novel framework for optimizing lighting configurations in photometric stereo. By discretizing the light space and using a learned network (LSNet), LIPIDS efficiently finds minimal, universal lighting setups that improve surface normal estimation accuracy across various photometric stereo methods and datasets.

What Problem Does This Paper Address?

Photometric stereo is a powerful technique for reconstructing 3D surface shapes by analyzing how an object's appearance changes under different lighting conditions. However, traditional methods often require a large number of images captured under carefully orchestrated, often sparsely sampled, lighting conditions. This can be impractical due to time, cost, and equipment limitations. Furthermore, existing datasets might not cover the full spectrum of lighting variations, leading to suboptimal results. The core challenge this paper addresses is how to intelligently select an optimal, minimal set of lighting configurations that can generalize across different objects and photometric stereo pipelines, thereby making the process more efficient and effective.

Methods

The proposed method, LIPIDS (Learning-based Illumination Planning In Discretized Light Space), tackles this problem in two main stages:

  1. Light-space Discretization: The continuous light space (upper hemisphere) is first discretized into a fixed number of bins (K=48 in their experiments). This simplifies the problem from selecting from infinite possibilities to selecting from a finite set of regions.
  2. Light Sampling Network (LSNet): A novel neural network, LSNet, is introduced to learn an optimal selection of M lights from the discretized bins. LSNet uses a learnable weight matrix that, through a softmax operation and an increasing temperature parameter during training, encourages sparsity, effectively selecting a minimal set of light bins. This network is trained in conjunction with a Normal Regression Network (NRNet), which estimates surface normals from the selected lights. The training objective is to minimize the difference between the estimated and ground truth surface normals.

The key novelty lies in learning to discretize and select light directions in an offline manner, aiming for universal configurations that can be applied to any photometric stereo pipeline without requiring per-object optimization during inference.

Datasets

The authors utilize the following datasets:

  • Synthetic: Blobby & Sculpture dataset [6] for training the LSNet and NRNet. This dataset is noted for its near-uniform lighting distribution over the upper hemisphere.
  • Real-world:
    • DiLiGenT dataset [25]
    • DiLiGenT102 dataset [23]
    • Gourd & Apple dataset [2]
    • Light Stage Data Gallery [7]

These datasets are used to evaluate the performance of LIPIDS when integrated with existing photometric stereo backbones.

Key Insights and Ideas

  • Discretization is Key: Treating the light space as discrete bins, rather than exact directions, simplifies the optimization problem and allows for more generalizable solutions.
  • Learning Optimal Light Selection: Instead of relying on heuristic or exhaustive methods, learning an optimal subset of lights through LSNet significantly improves efficiency and accuracy.
  • Universality Matters: The goal is to find lighting configurations that work well across different objects and photometric stereo methods, reducing the need for specialized setups for each scenario.
  • Minimal Lights for Maximum Gain: LIPIDS demonstrates that a relatively small number of well-chosen lights can achieve performance comparable to or better than methods using many more lights.
  • Offline Planning for Efficiency: By performing illumination planning offline, LIPIDS enables faster inference times compared to online, per-object optimization methods.

Key Results

  • LIPIDS consistently outperforms or closely matches existing illumination planning methods (DC, TK, ReLeaPS) across various datasets and photometric stereo backbones (LS, PS-FCN, CNN-PS).
  • For instance, on the DiLiGenT dataset, LIPIDS showed a strong lead when averaged across different backbones and configurations.
  • The method demonstrates that a minimal number of lights (e.g., 10 or 20) can be sufficient for good performance, with performance saturating beyond 20 lights for some datasets.
  • Qualitative results show improved surface normal estimation, particularly in regions affected by shadows and inter-reflections, when using LIPIDS-selected lighting.
  • When combined with the SDM-UniPS backbone, LIPIDS also showed improvements in normal estimation accuracy.

Limitations and Future Work

  • Limitations: The paper acknowledges that while their method is offline, the training process for LSNet and NRNet can take around 3 hours. While this is significantly faster than some online methods, it's still a considerable training time. The paper also notes that the performance on DiLiGenT102 slightly lagged behind some methods with 20 lights, possibly due to different training paradigms of the backbones.
  • Future Work: The authors express hope that the simplicity and applicability of LIPIDS will encourage further research into optimally minimizing photometric stereo through illumination planning. They suggest that the framework can be integrated with existing photometric stereo networks to improve performance, especially in sparse settings, by optimally utilizing existing datasets.

Summary updated

BibTeX
@inproceedings{tiwari2025lipids,
  author = {Ashish Tiwari and Mihirkumar Sutariya and Shanmuganathan Raman},
  title = {{LIPIDS: Learning-Based Illumination Planning in Discretized (Light) Space for Photometric Stereo}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure