Lighting estimation in fringe images during motion compensation for 3D measurements

Andreas Breitbarth, Peter Kühmstedt, Gunther Notni, Joachim DenzlerProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2013 · DOI

TL;DR

This paper introduces a novel method to accurately reconstruct 3D objects in motion by compensating for both object movement and changing lighting conditions. By estimating lighting variations per pixel, the approach enables robust 3D measurements even with dynamic scenes, overcoming a key limitation of traditional fringe projection techniques.

What Problem Does This Paper Address?

Traditional fringe projection, a powerful technique for 3D shape measurement, relies on the assumption that the object and camera remain static during the measurement sequence. This assumption breaks down when dealing with moving objects or dynamic sensor setups (like handheld scanners). In such scenarios, the correspondence between 3D object points and their 2D projections in camera images becomes unstable. While previous work has addressed motion compensation by realigning images, a significant challenge remains: changing lighting conditions. These variations, which can occur due to object movement or external light sources, introduce distortions that corrupt the phase information crucial for accurate 3D reconstruction. This paper tackles the problem of estimating and compensating for these per-pixel lighting changes in the context of motion-compensated 3D measurements.

Methods

The core of the proposed method lies in estimating lighting changes at a per-pixel level. The authors build upon existing phase-shifting fringe projection techniques, which typically model intensity as a function of additive and multiplicative distortions and phase. For static scenes, these distortions are assumed constant. However, for dynamic scenes with motion, these parameters become variable.

The paper proposes an approach that:

  1. Motion Compensation: First, it aligns acquired fringe images to compensate for object motion. This step is crucial for re-establishing stable correspondences.
  2. Lighting Estimation: For each pixel, the method estimates local additive ($a(x, y)$) and multiplicative ($b(x, y)$) lighting distortions. This is achieved by analyzing the intensity values and phase shifts within a spatial neighborhood of pixels across multiple time steps. The estimation leverages a system of equations derived from the fringe projection model, adapted for dynamic scenarios.
  3. Phase Calculation: Using the estimated lighting parameters and motion-compensated images, the method calculates accurate phase values.
  4. 3D Reconstruction: Finally, these corrected phase values are used to reconstruct a dense 3D point cloud.

A key aspect is the use of an adaptive kernel size for lighting estimation, allowing for a trade-off between noise reduction in low-contrast areas and preserving detail at object edges.

Datasets

The authors primarily use simulated data for evaluation. This includes:

  • Simulated fringe images of a rotating prism with varying lighting conditions (e.g., light fall-off).
  • Simulated fringe images with added Gaussian noise to test robustness.
  • Simulated fringe images with shades introduced by other objects to evaluate performance in complex scenes.

They also mention performing one experiment with real input data using a prism, where the ground truth 3D data was obtained from a previous static measurement.

Key Insights and Ideas

  • Per-pixel lighting estimation is critical for dynamic 3D measurements: The paper demonstrates that simply compensating for motion is insufficient if lighting changes are not accounted for, especially for non-translational movements.
  • Adaptive kernel size is crucial for robust lighting estimation: The choice of neighborhood size for lighting estimation significantly impacts the results. A larger kernel smooths out noise but can blur edges, while a smaller kernel preserves edges but can be noisy in low-contrast regions. An adaptive approach is suggested to balance these trade-offs.
  • Lighting changes have a more pronounced effect on non-translational motion: The authors note that for simple translational motion, lighting changes have a less severe impact, but for arbitrary rotations, their compensation method becomes essential.

Key Results

  • The proposed method successfully estimates lighting changes per pixel, leading to improved 3D reconstruction accuracy for moving objects.
  • Simulations show that the method achieves high completeness (e.g., >98% for many motion types) and low average 3D errors, often in the range of tens to a few hundred micrometers for a 180mm prism.
  • The method demonstrates robustness to Gaussian noise, with noticeable degradation only at higher noise levels (RMS > 5).
  • Experiments with simulated shades indicate that while most of the object is reconstructed correctly, errors can occur near the boundaries of these shades.

Limitations and Future Work

  • Shaded regions: The paper acknowledges that errors can occur in regions with shades introduced by other objects, particularly near the shade borders. This is identified as a challenging area requiring further research.
  • Computational time: The lighting estimation part of the algorithm is computationally intensive, taking approximately one minute on a modern CPU core. Reducing this calculation time is suggested as a future improvement.
  • Edge detail: While the adaptive kernel helps, there's still a trade-off between smoothness and edge detail, suggesting further investigation into optimizing kernel selection or alternative edge-handling techniques.

Summary updated

Abstract

Fringe projection is an established method to measure the 3D structure of macroscopic objects. To achieve both a high accuracy and robustness a certain number of images with pairwise different projection pattern is required. Over this sequence it is necessary that each 3D object point corresponds to the same image point at every time. This situation is no longer given for measurements under motion. One possibility to solve this problem is to restore the static situation. Therefore, the acquired camera images have to be realigned and secondly, the degree of fringe shift has to be estimated. Furthermore, there exists another variable: change in lighting. The compensation of these variances is a difficult task and could only be realized with several assumptions, but it has to be approximately determined and integrated into the 3D reconstruction process. We propose a method to estimate these lighting changes for each camera pixel with respect to their neighbors at each point in time. The algorithms were validated on simulation data, in particular with rotating measurement objects. For translational motion, lighting changes have no severe effect in our applications. Taken together, without using high-speed hardware our method results in a motion compensated dense 3D point cloud which is eligible for three-dimensional measurement of moving objects or setups with sensor systems in motion.
BibTeX
@inproceedings{breitbarth2013lighting,
  author = {Andreas Breitbarth and Peter Kühmstedt and Gunther Notni and Joachim Denzler},
  title = {{Lighting estimation in fringe images during motion compensation for 3D measurements}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2013},
  doi = {10.1117/12.2020500},
}
Figure