Physics-Free Spectrally Multiplexed Photometric Stereo under Unknown Spectral Composition

Satoshi Ikehata, Yuta Asano

Abstract

SpectraM-PS is a novel deep learning framework designed to recover high-quality surface normals from a single "spectrally multiplexed" image. Unlike traditional methods, it operates without requiring calibrated lights or sensors. By transforming "spectral ambiguity" from a mathematical hurdle into a training advantage, SpectraM-PS enables high-fidelity dynamic surface recovery in uncalibrated, real-world settings.

The Challenge of Dynamic Photometric Stereo

Photometric Stereo (PS) is a cornerstone technique in 3D reconstruction, estimating surface normals by capturing multiple images of an object under varying lighting directions. While accurate, traditional PS is limited to static objects. To address dynamic scenes, researchers employ "spectral multiplexing," where different colored lights are shone simultaneously and captured in separate camera channels (e.g., RGB).

However, existing multiplexed PS methods are notoriously brittle. They typically require precise calibration of light directions, light spectra, and sensor sensitivities. Furthermore, they struggle with "crosstalk"—where light from one color channel bleeds into another—and often assume objects possess uniform materials. SpectraM-PS aims to overcome these constraints, enabling dynamic surface recovery using off-the-shelf, uncalibrated equipment.

SpectraM-PS provides a robust pipeline for mapping spectrally multiplexed images to normal maps under uncontrolled lighting and sensor conditions, even for dynamic video streams.

Methods: A Physics-Free Approach

The authors introduce SpectraM-PS, a "physics-free" neural network that learns to map multiplexed images directly to surface normals without relying on explicit physical models.

Transformer-based Architecture

The model utilizes Transformers to process input channels, making it "order-agnostic." This flexibility allows the network to handle any number of spectral bands and various sensor types (RGB, multispectral, NIR) without retraining.

Dual-scale Decoder

To capture both the overall shape and fine surface details, the decoder operates at two scales: a global scale for low-frequency shape and a local "patch-embedding" scale to recover high-frequency textures.

The SpectraM-PS architecture features a dual-scale approach with global and local branches to capture both coarse shape and fine surface details.

Leveraging Spectral Ambiguity

A key theoretical insight is that "spectral ambiguity"—the fact that many different combinations of light, surface, and sensor can produce the same pixel value—actually "compacts" the input space. This allows the authors to train their model effectively on synthetic data generated with standard RGB rendering rather than needing complex multispectral assets.

Datasets and Benchmarking

  • PS-Multiplex (Training): A synthetic dataset of over 106,000 multiplexed images rendered in Blender, featuring 410 3D models with diverse materials (diffuse, specular, metallic).
  • SpectraM14 (Benchmark): The first real-world benchmark for spectrally multiplexed PS, featuring 14 objects captured under five distinct conditions, ranging from ideal to strong crosstalk and Near-Infrared sensing.

Key Results

SpectraM-PS significantly outperforms existing state-of-the-art (SOTA) calibrated methods across all benchmark conditions:

  • Superior Accuracy: In "weak crosstalk" conditions, SpectraM-PS achieved a Mean Angular Error (MAE) of 10.0°, nearly doubling the accuracy of the best calibrated competitor (21.7°).
  • Robustness to Crosstalk: Even under "strong crosstalk," the model maintained high accuracy (11.6° MAE), while traditional methods saw error rates skyrocket to over 50°.
  • Dynamic Capability: The method successfully recovers stable normal maps from video streams of moving objects, demonstrating its utility for real-world dynamic scene analysis.

Limitations and Future Work

While SpectraM-PS represents a significant leap forward, challenges remain. Currently, the model takes several seconds to process a single frame, making inference acceleration a priority for industrial applications. Additionally, while the results are impressive, there is some temporal "jitter" in video frames; future work may incorporate temporal consistency constraints. Finally, like most PS methods, performance can degrade in areas with heavy cast shadows common in complex dynamic scenes.

Summary updated

BibTeX
@inproceedings{ikehata2024physics,
  author = {Satoshi Ikehata and Yuta Asano},
  title = {{Physics-Free Spectrally Multiplexed Photometric Stereo under Unknown Spectral Composition}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
}
Figure