Abstract
SpectraM-PS is a novel deep learning framework designed to recover high-quality surface normals from a single "spectrally multiplexed" image. Unlike traditional methods, it operates without requiring calibrated lights or sensors. By transforming "spectral ambiguity" from a mathematical hurdle into a training advantage, SpectraM-PS enables high-fidelity dynamic surface recovery in uncalibrated, real-world settings.
The Challenge of Dynamic Photometric Stereo
Photometric Stereo (PS) is a cornerstone technique in 3D reconstruction, estimating surface normals by capturing multiple images of an object under varying lighting directions. While accurate, traditional PS is limited to static objects. To address dynamic scenes, researchers employ "spectral multiplexing," where different colored lights are shone simultaneously and captured in separate camera channels (e.g., RGB).
However, existing multiplexed PS methods are notoriously brittle. They typically require precise calibration of light directions, light spectra, and sensor sensitivities. Furthermore, they struggle with "crosstalk"—where light from one color channel bleeds into another—and often assume objects possess uniform materials. SpectraM-PS aims to overcome these constraints, enabling dynamic surface recovery using off-the-shelf, uncalibrated equipment.
Methods: A Physics-Free Approach
The authors introduce SpectraM-PS, a "physics-free" neural network that learns to map multiplexed images directly to surface normals without relying on explicit physical models.
Transformer-based Architecture
The model utilizes Transformers to process input channels, making it "order-agnostic." This flexibility allows the network to handle any number of spectral bands and various sensor types (RGB, multispectral, NIR) without retraining.
Dual-scale Decoder
To capture both the overall shape and fine surface details, the decoder operates at two scales: a global scale for low-frequency shape and a local "patch-embedding" scale to recover high-frequency textures.
Leveraging Spectral Ambiguity
A key theoretical insight is that "spectral ambiguity"—the fact that many different combinations of light, surface, and sensor can produce the same pixel value—actually "compacts" the input space. This allows the authors to train their model effectively on synthetic data generated with standard RGB rendering rather than needing complex multispectral assets.
Datasets and Benchmarking
- PS-Multiplex (Training): A synthetic dataset of over 106,000 multiplexed images rendered in Blender, featuring 410 3D models with diverse materials (diffuse, specular, metallic).
- SpectraM14 (Benchmark): The first real-world benchmark for spectrally multiplexed PS, featuring 14 objects captured under five distinct conditions, ranging from ideal to strong crosstalk and Near-Infrared sensing.
Key Results
SpectraM-PS significantly outperforms existing state-of-the-art (SOTA) calibrated methods across all benchmark conditions:
- Superior Accuracy: In "weak crosstalk" conditions, SpectraM-PS achieved a Mean Angular Error (MAE) of 10.0°, nearly doubling the accuracy of the best calibrated competitor (21.7°).
- Robustness to Crosstalk: Even under "strong crosstalk," the model maintained high accuracy (11.6° MAE), while traditional methods saw error rates skyrocket to over 50°.
- Dynamic Capability: The method successfully recovers stable normal maps from video streams of moving objects, demonstrating its utility for real-world dynamic scene analysis.
Limitations and Future Work
While SpectraM-PS represents a significant leap forward, challenges remain. Currently, the model takes several seconds to process a single frame, making inference acceleration a priority for industrial applications. Additionally, while the results are impressive, there is some temporal "jitter" in video frames; future work may incorporate temporal consistency constraints. Finally, like most PS methods, performance can degrade in areas with heavy cast shadows common in complex dynamic scenes.
Summary updated