Near-Light Color Photometric Stereo for Mono-Chromatic Non-Lambertian Surfaces

Zonglin Li, Jieji Ren, Shuangfan Zhou, Heng Guo, Jinnuo Zhang, Jiang Zhou, Boxin Shi, Zhanyu Ma, Guoying Gu

TL;DR

Reconstructing detailed 3D surfaces from a single color image typically requires assuming distant lighting and perfectly matte surfaces. This paper introduces a self-supervised framework that leverages neural implicit representations to achieve accurate, single-shot 3D reconstruction under realistic near-light conditions for shiny, uniform-color (mono-chromatic) objects. The authors successfully validate their approach using both synthetic data and a custom-built robotic tactile sensor.

What Problem Does This Paper Address?

Photometric Stereo (PS) is a classic computer vision technique that estimates the 3D shape of an object by analyzing how its appearance changes under different lighting directions. To capture moving objects, Color Photometric Stereo (CPS) uses a clever trick: it shines red, green, and blue lights from different angles simultaneously, allowing a 3D shape to be recovered from a single "single-shot" RGB image.

However, traditional CPS relies on two idealistic assumptions:

  1. Distant Lighting: Light rays are assumed to be parallel and uniform.
  2. Lambertian Reflectance: The surface must be perfectly matte, reflecting light equally in all directions.

In compact, real-world setups—such as robotic tactile sensors—these assumptions fall apart. The lights are close to the object (near-light conditions), meaning light intensity drops off sharply with distance. Furthermore, real-world materials are often glossy or metallic (non-Lambertian). This combination makes the problem highly ill-posed because a single pixel does not provide enough independent observations to solve for both the complex material reflections and the surface orientation.

Methods

To make this underdetermined problem solvable, the authors introduce a self-supervised framework that combines neural implicit representations with a mono-chromaticity prior (assuming the object has a uniform color and homogeneous material composition).

Overview of the proposed framework, taking a single RGB image and light positions to output mesh, depth, and normals.

The pipeline consists of two primary neural components:

  1. Neural Surface Model: Instead of predicting surface normals directly, this network uses a multi-resolution hash encoding paired with a sinusoidal MLP (SIREN) to estimate per-pixel depth values ($z$). The surface normals ($n$) are then analytically derived from the spatial gradients of this depth map using a perspective camera model. This mathematically couples depth and normals, ensuring geometric consistency.
  2. Neural BRDF Model: To handle shiny, non-matte materials, a separate neural network models the object's Bidirectional Reflectance Distribution Function (BRDF). It maps the angles between the surface normal, light source, and camera view to a reflectance value ($r$), utilizing independent MLP branches for the R, G, and B channels to capture wavelength-dependent reflections.

The entire system is trained end-to-end using Differentiable Rendering. The network renders a synthetic image based on its current shape and material estimates, accounts for near-light attenuation, and minimizes the difference ($\ell_1$ loss) against the captured image. No ground-truth 3D shapes are required for training.

Additionally, because compact hardware suffers from color bleeding between camera filters, the authors introduce a lightweight Crosstalk Correction Module (CCM) to clean up the RGB channels before shape estimation.

Datasets

The method was evaluated using two primary datasets:

  • Synthetic Dataset: Because no public datasets exist for near-light CPS, the authors used Blender to render five distinct objects (White gypsum, Brown metal, Brown gypsum, White ceramic, Green ceramic). These objects covered a wide range of geometric complexities and material types (matte, metallic, and ceramic).
  • Real-World Data: Collected using a custom-engineered, compact visuo-tactile sensor named NearLightTactile, which uses an elastomer interface under single-shot colored illumination to capture sub-millimeter surface textures.

Key Insights and Ideas

  • Depth-First Normal Derivation: Directly predicting surface normals in near-light conditions often leads to physically impossible shapes due to geometric decoupling. Deriving normals analytically from an underlying depth network inherently enforces valid 3D geometry.
  • The Power of Mono-Chromaticity: Assuming a uniform material means that the complex BRDF is identical across the entire surface. This allows a single, global neural network to learn the material properties effectively, transforming an otherwise impossible single-pixel problem into a well-posed global optimization problem.
  • Perfect Fit for Tactile Sensing: While the mono-chromaticity assumption might seem restrictive for general scenes, it perfectly matches the physics of elastomer-based robotic tactile sensors (like GelSight), making this method highly practical for robotic perception.

Key Results

The proposed method was compared against several state-of-the-art photometric stereo techniques using Mean Angular Error (MAE) in degrees for the estimated surface normals.

Quantitative Performance

As shown in the paper's evaluation, distant-light methods (LSPS, RGBPS) and standard near-light methods (LEDPS) struggled significantly with non-matte objects.

  • On the shiny Brown metal object, the proposed method achieved an MAE of 4.74°, drastically outperforming LEDPS (13.11°) and LSPS (24.67°).
  • On the Brown gypsum object, the method achieved its best performance with an MAE of 2.43°.

Qualitative Performance

In real-world testing with the NearLightTactile sensor, the framework successfully recovered high-fidelity, sub-millimeter 3D structures from a single image, demonstrating excellent robustness against color crosstalk.

Limitations and Future Work

While powerful, the method has a few notable limitations:

  • Mono-Chromatic Constraint: The framework relies strictly on the assumption that the target object is made of a single, uniformly colored material. It cannot currently reconstruct multi-colored or multi-material surfaces.
  • Calibrated Lighting: The system requires prior knowledge of the exact positions of the near-light sources, meaning it still depends on an initial hardware calibration step.

Future work could involve extending the framework to handle uncalibrated lighting or adapting the neural BRDF model to accommodate multi-material objects.

Summary updated

Abstract

Color photometric stereo enables single-shot surface reconstruction, extending conventional photometric stereo that requires multiple images of a static scene under varying illumination to dynamic scenarios. However, most existing approaches assume ideal distant lighting and Lambertian reflectance, leaving more practical near-light conditions and non-Lambertian surfaces underexplored. To overcome this limitation, we propose a framework that leverages neural implicit representations for depth and BRDF modeling under the assumption of mono-chromaticity (uniform chromaticity and homogeneous material), which alleviates the inherent ill-posedness of color photometric stereo and allows for detailed surface recovery from just one image. Furthermore, we design a compact optical tactile sensor to validate our approach. Experiments on both synthetic and real-world datasets demonstrate that our method achieves accurate and robust surface reconstruction.
BibTeX
@misc{li2026near,
  author = {Zonglin Li and Jieji Ren and Shuangfan Zhou and Heng Guo and Jinnuo Zhang and Jiang Zhou and Boxin Shi and Zhanyu Ma and Guoying Gu},
  title = {{Near-Light Color Photometric Stereo for Mono-Chromatic Non-Lambertian Surfaces}},
  year = {2026},
  eprint = {2601.12666},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.12666},
}
Figure