Abstract
This paper presents a new algorithm for enforcing temporal coherence on depth estimation from multi-view videos of dynamic scenes as well as the first substantial quantitative evaluation of the improvement in depth estimation accuracy due to temporal coherence. The proposed algorithm is generally applicable and practical since it bypasses explicit scene flow estimation, which has a very large state space, and relies only on optical flow which is used to impose soft constraints on depth estimation for the next frame. As a result, our algorithm is applicable to scenes with large depth and motion ranges. The output is a sequence of depth maps that can be used for novel view synthesis among other applications. While it is intuitive that enforcing temporal coherence should improve the accuracy of depth estimation, this improvement has never been assessed quantitatively due to the lack of data with ground truth. To overcome this limitation we use the image prediction error as the criterion and show that the benefits of temporal coherence are significant on a diverse set of multi-view video sequences.
Introduction
While 3D reconstruction of static scenes has reached near-photorealistic quality, dynamic scenes—where objects move independently of the camera—remain a significant challenge. Most current methods reconstruct each video frame independently, ignoring the temporal continuity inherent in video. While it is intuitive that information from previous frames should help refine the current frame's depth map, quantifying this improvement is difficult because there is almost no "ground truth" data for moving 3D scenes.
This research addresses two main hurdles: the computational complexity of estimating 3D "scene flow" and the lack of evaluation metrics for dynamic depth estimation.
Methodology
The authors propose a two-stage approach that avoids explicit scene flow estimation, making it practical for real-world applications.
Spatial Matching
The process begins with a plane-sweeping algorithm to compute the initial likelihood of depth for each pixel. This is based on photoconsistency across multiple synchronized cameras, establishing a baseline for spatial accuracy.
Temporal Constraints
Instead of hard-coding complex 3D velocities, the algorithm leverages optical flow (2D pixel motion) to predict where a pixel moves in subsequent frames. The core idea is to treat temporal coherence as a "soft constraint": the depth of a pixel in a new frame is assumed to be close to its depth in the previous frame, modeled by a Gaussian probability distribution.
Optimization
Spatial and temporal costs are integrated into a single cost volume. The algorithm then employs Semi-Global Matching (SGM) to find optimal depth assignments that maintain both spatial smoothness and temporal consistency.
Evaluation via View Prediction
To circumvent the lack of ground truth depth, the researchers use view prediction error. By withholding data from one camera and using the estimated depth maps to "predict" its view, they can measure the RGB difference between the prediction and the actual captured image to quantify accuracy.
Datasets and Experimental Setup
The algorithm was evaluated on four diverse multi-view video datasets:
- Cheongsam: 20 cameras in a 4.2m dome.
- Ballet: 8 cameras in a 30° arc.
- Book Arrival: 16-camera rig in a side-by-side parallel configuration.
- Outdoor: 16-camera rig in a side-by-side parallel configuration.
Results and Discussion
The quantitative results confirm that temporal coherence significantly enhances depth estimation accuracy.
Accuracy Gains
In the "Book Arrival" dataset, the average RGB error dropped from 85.9 to 61.5, representing a nearly 30% improvement in reconstruction accuracy. Furthermore, the percentage of pixels that could be correctly projected to a new view increased across all datasets (e.g., from 90.4% to 93.7% for "Book Arrival").
Qualitative Improvements
Qualitatively, the resulting depth maps are much cleaner, particularly on stationary backgrounds and moving foreground objects. The synthesized videos exhibit significantly fewer "flickering" artifacts compared to frame-by-frame reconstructions.
Key Insights
- Soft vs. Hard Constraints: Using soft probabilities rather than hard pixel-to-pixel correspondences makes the algorithm robust to optical flow errors and capable of handling large motion ranges.
- Optimal Time Horizons: Testing different "horizons" revealed that accuracy peaks at approximately 3 frames. Looking back further can lead to error propagation and "blurred" surfaces.
- Efficiency: Bypassing 3D scene flow in favor of 2D optical flow provides a significant practical shortcut, making the method applicable to scenes with large depth ranges.
Conclusion
This work demonstrates that enforcing temporal coherence is a major "win" for dynamic depth estimation. While the method relies on the accuracy of the underlying optical flow and faces challenges in textureless regions, the quantitative gains are substantial. Future work may explore how camera configurations and image frequency content further influence the quality of novel view synthesis.
Summary updated