TL;DR
Reconstructing detailed 3D surfaces from a single color image typically requires assuming distant lighting and perfectly matte surfaces. This paper introduces a self-supervised framework that leverages neural implicit representations to achieve accurate, single-shot 3D reconstruction under realistic near-light conditions for shiny, uniform-color (mono-chromatic) objects. The authors successfully validate their approach using both synthetic data and a custom-built robotic tactile sensor.
What Problem Does This Paper Address?
Photometric Stereo (PS) is a classic computer vision technique that estimates the 3D shape of an object by analyzing how its appearance changes under different lighting directions. To capture moving objects, Color Photometric Stereo (CPS) uses a clever trick: it shines red, green, and blue lights from different angles simultaneously, allowing a 3D shape to be recovered from a single "single-shot" RGB image.
However, traditional CPS relies on two idealistic assumptions:
- Distant Lighting: Light rays are assumed to be parallel and uniform.
- Lambertian Reflectance: The surface must be perfectly matte, reflecting light equally in all directions.
In compact, real-world setups—such as robotic tactile sensors—these assumptions fall apart. The lights are close to the object (near-light conditions), meaning light intensity drops off sharply with distance. Furthermore, real-world materials are often glossy or metallic (non-Lambertian). This combination makes the problem highly ill-posed because a single pixel does not provide enough independent observations to solve for both the complex material reflections and the surface orientation.
Methods
To make this underdetermined problem solvable, the authors introduce a self-supervised framework that combines neural implicit representations with a mono-chromaticity prior (assuming the object has a uniform color and homogeneous material composition).
The pipeline consists of two primary neural components:
- Neural Surface Model: Instead of predicting surface normals directly, this network uses a multi-resolution hash encoding paired with a sinusoidal MLP (SIREN) to estimate per-pixel depth values ($z$). The surface normals ($n$) are then analytically derived from the spatial gradients of this depth map using a perspective camera model. This mathematically couples depth and normals, ensuring geometric consistency.
- Neural BRDF Model: To handle shiny, non-matte materials, a separate neural network models the object's Bidirectional Reflectance Distribution Function (BRDF). It maps the angles between the surface normal, light source, and camera view to a reflectance value ($r$), utilizing independent MLP branches for the R, G, and B channels to capture wavelength-dependent reflections.
The entire system is trained end-to-end using Differentiable Rendering. The network renders a synthetic image based on its current shape and material estimates, accounts for near-light attenuation, and minimizes the difference ($\ell_1$ loss) against the captured image. No ground-truth 3D shapes are required for training.
Additionally, because compact hardware suffers from color bleeding between camera filters, the authors introduce a lightweight Crosstalk Correction Module (CCM) to clean up the RGB channels before shape estimation.
Datasets
The method was evaluated using two primary datasets:
- Synthetic Dataset: Because no public datasets exist for near-light CPS, the authors used Blender to render five distinct objects (White gypsum, Brown metal, Brown gypsum, White ceramic, Green ceramic). These objects covered a wide range of geometric complexities and material types (matte, metallic, and ceramic).
- Real-World Data: Collected using a custom-engineered, compact visuo-tactile sensor named NearLightTactile, which uses an elastomer interface under single-shot colored illumination to capture sub-millimeter surface textures.
Key Insights and Ideas
- Depth-First Normal Derivation: Directly predicting surface normals in near-light conditions often leads to physically impossible shapes due to geometric decoupling. Deriving normals analytically from an underlying depth network inherently enforces valid 3D geometry.
- The Power of Mono-Chromaticity: Assuming a uniform material means that the complex BRDF is identical across the entire surface. This allows a single, global neural network to learn the material properties effectively, transforming an otherwise impossible single-pixel problem into a well-posed global optimization problem.
- Perfect Fit for Tactile Sensing: While the mono-chromaticity assumption might seem restrictive for general scenes, it perfectly matches the physics of elastomer-based robotic tactile sensors (like GelSight), making this method highly practical for robotic perception.
Key Results
The proposed method was compared against several state-of-the-art photometric stereo techniques using Mean Angular Error (MAE) in degrees for the estimated surface normals.
Quantitative Performance
As shown in the paper's evaluation, distant-light methods (LSPS, RGBPS) and standard near-light methods (LEDPS) struggled significantly with non-matte objects.
- On the shiny Brown metal object, the proposed method achieved an MAE of 4.74°, drastically outperforming LEDPS (13.11°) and LSPS (24.67°).
- On the Brown gypsum object, the method achieved its best performance with an MAE of 2.43°.
Qualitative Performance
In real-world testing with the NearLightTactile sensor, the framework successfully recovered high-fidelity, sub-millimeter 3D structures from a single image, demonstrating excellent robustness against color crosstalk.
Limitations and Future Work
While powerful, the method has a few notable limitations:
- Mono-Chromatic Constraint: The framework relies strictly on the assumption that the target object is made of a single, uniformly colored material. It cannot currently reconstruct multi-colored or multi-material surfaces.
- Calibrated Lighting: The system requires prior knowledge of the exact positions of the near-light sources, meaning it still depends on an initial hardware calibration step.
Future work could involve extending the framework to handle uncalibrated lighting or adapting the neural BRDF model to accommodate multi-material objects.
Summary updated