TL;DR
This paper introduces a novel method to accurately reconstruct 3D objects in motion by compensating for both object movement and changing lighting conditions. By estimating lighting variations per pixel, the approach enables robust 3D measurements even with dynamic scenes, overcoming a key limitation of traditional fringe projection techniques.
What Problem Does This Paper Address?
Traditional fringe projection, a powerful technique for 3D shape measurement, relies on the assumption that the object and camera remain static during the measurement sequence. This assumption breaks down when dealing with moving objects or dynamic sensor setups (like handheld scanners). In such scenarios, the correspondence between 3D object points and their 2D projections in camera images becomes unstable. While previous work has addressed motion compensation by realigning images, a significant challenge remains: changing lighting conditions. These variations, which can occur due to object movement or external light sources, introduce distortions that corrupt the phase information crucial for accurate 3D reconstruction. This paper tackles the problem of estimating and compensating for these per-pixel lighting changes in the context of motion-compensated 3D measurements.
Methods
The core of the proposed method lies in estimating lighting changes at a per-pixel level. The authors build upon existing phase-shifting fringe projection techniques, which typically model intensity as a function of additive and multiplicative distortions and phase. For static scenes, these distortions are assumed constant. However, for dynamic scenes with motion, these parameters become variable.
The paper proposes an approach that:
- Motion Compensation: First, it aligns acquired fringe images to compensate for object motion. This step is crucial for re-establishing stable correspondences.
- Lighting Estimation: For each pixel, the method estimates local additive ($a(x, y)$) and multiplicative ($b(x, y)$) lighting distortions. This is achieved by analyzing the intensity values and phase shifts within a spatial neighborhood of pixels across multiple time steps. The estimation leverages a system of equations derived from the fringe projection model, adapted for dynamic scenarios.
- Phase Calculation: Using the estimated lighting parameters and motion-compensated images, the method calculates accurate phase values.
- 3D Reconstruction: Finally, these corrected phase values are used to reconstruct a dense 3D point cloud.
A key aspect is the use of an adaptive kernel size for lighting estimation, allowing for a trade-off between noise reduction in low-contrast areas and preserving detail at object edges.
Datasets
The authors primarily use simulated data for evaluation. This includes:
- Simulated fringe images of a rotating prism with varying lighting conditions (e.g., light fall-off).
- Simulated fringe images with added Gaussian noise to test robustness.
- Simulated fringe images with shades introduced by other objects to evaluate performance in complex scenes.
They also mention performing one experiment with real input data using a prism, where the ground truth 3D data was obtained from a previous static measurement.
Key Insights and Ideas
- Per-pixel lighting estimation is critical for dynamic 3D measurements: The paper demonstrates that simply compensating for motion is insufficient if lighting changes are not accounted for, especially for non-translational movements.
- Adaptive kernel size is crucial for robust lighting estimation: The choice of neighborhood size for lighting estimation significantly impacts the results. A larger kernel smooths out noise but can blur edges, while a smaller kernel preserves edges but can be noisy in low-contrast regions. An adaptive approach is suggested to balance these trade-offs.
- Lighting changes have a more pronounced effect on non-translational motion: The authors note that for simple translational motion, lighting changes have a less severe impact, but for arbitrary rotations, their compensation method becomes essential.
Key Results
- The proposed method successfully estimates lighting changes per pixel, leading to improved 3D reconstruction accuracy for moving objects.
- Simulations show that the method achieves high completeness (e.g., >98% for many motion types) and low average 3D errors, often in the range of tens to a few hundred micrometers for a 180mm prism.
- The method demonstrates robustness to Gaussian noise, with noticeable degradation only at higher noise levels (RMS > 5).
- Experiments with simulated shades indicate that while most of the object is reconstructed correctly, errors can occur near the boundaries of these shades.
Limitations and Future Work
- Shaded regions: The paper acknowledges that errors can occur in regions with shades introduced by other objects, particularly near the shade borders. This is identified as a challenging area requiring further research.
- Computational time: The lighting estimation part of the algorithm is computationally intensive, taking approximately one minute on a modern CPU core. Reducing this calculation time is suggested as a future improvement.
- Edge detail: While the adaptive kernel helps, there's still a trade-off between smoothness and edge detail, suggesting further investigation into optimizing kernel selection or alternative edge-handling techniques.
Summary updated