Abstract
The availability of high-speed 3D video sensors has greatly facilitated 3D shape acquisition of dynamic and deformable objects, but high frame rate 3D reconstruction is always degraded by spatial noise and temporal fluctuations. This paper presents a simple yet powerful intensity video guided multi-frame 4D fusion pipeline. Temporal tracking of intensity image points (of moving and deforming objects) allows registration of the corresponding 3D data points, whose 3D noise and fluctuations are then reduced by spatio-temporal multi-frame 4D fusion. We conducted simulated noise tests and real experiments on four 3D objects using a 1000 fps 3D video sensor. The results demonstrate that the proposed algorithm is effective at reducing 3D noise and is robust against intensity noise. It outperforms existing algorithms with good scalability on both stationary and dynamic objects.
Introduction
Capturing the 3D shape of moving or deforming objects—such as a speaking face or a bouncing ball—requires high-speed sensors. However, as frame rates increase (e.g., to 1000 fps), the quality of individual 3D frames often decreases due to spatial noise and temporal fluctuations. Surfaces appear "rough," and data flickers over time. Existing denoising methods often struggle with deformable objects; simple frame averaging causes motion blur, while standard filters can lead to loss of geometric detail or "shrinking" of the object's dimensions.
Proposed Methodology
The authors propose a two-stage "4D Fusion" pipeline that integrates 3D spatial data with temporal information guided by 2D intensity video.
Stage 1: Intensity-Guided 3D Motion Estimation
Rather than tracking motion in noisy 3D data, the system utilizes the 2D intensity (grayscale) video. A "Particle Belief Propagation" algorithm identifies pixel movement between frames. Since 2D images typically offer clearer texture and lower noise than depth maps, tracking is significantly more reliable. This 2D motion is subsequently projected into 3D space using a projective camera model.
Stage 2: Spatio-Temporal 4D Fusion
With the motion of every point established, the system aligns 3D points from multiple consecutive frames (typically 9). A bilateral-style filter is applied, considering spatial distance, intensity similarity, and temporal distance. This process fuses the data, averaging out noise while maintaining sharp features. A final hole-filling step uses gradient-directed interpolation to repair any surfaces where motion tracking may have failed.
Experimental Results
The method was evaluated using synthetic data (a falling ball with Gaussian noise) and real-world captures from a 1000 fps DI4D system, including a static textured plane, a human hand, and a speaking face.
Performance and Robustness
A key challenge in denoising is the "smoothing vs. shrinking" trade-off. As shown in Figure 6(a), standard methods like DNBF can over-smooth, leading to a loss of volume or "shrinking." The proposed 4D fusion achieves lower mean roughness (smoother surfaces) while maintaining high "shape correctness" (global dimensional accuracy), as illustrated in Figure 6(b).
On a static plane, the method reduced mean roughness from 0.62mm in raw data to 0.22mm. Furthermore, the algorithm demonstrated high resilience to noise in the intensity video, confirming the robustness of the motion estimation stage.
Conclusion
The intensity-guided 4D fusion pipeline effectively addresses the noise challenges inherent in high-speed 3D reconstruction. By leveraging 2D texture for motion tracking, it provides a scalable solution for both stationary and highly dynamic deformable objects. Future work will focus on optimizing computational complexity—currently a bottleneck due to the Belief Propagation stage—and addressing potential occlusion issues in complex motion scenarios.
Summary updated