When deep learning models estimate depth from a single static image, they perform remarkably well. However, if you feed a video into these same models frame-by-frame, the result is a jarring, flickering mess.
This phenomenon—known as temporal depth inconsistency—manifests as rapid fluctuations in predicted depth values and geometric drift across consecutive frames. As noted by Wang et al. and Düzçeker et al., treating video frames as independent images ignores the continuous nature of time. This lack of coherence makes raw per-frame depth maps unusable for downstream applications like 3D scene reconstruction, novel view synthesis, and augmented reality.
The Root Cause: Scale and Shift Ambiguity
To understand why video depth flickers, we have to look at the fundamental physics of a camera. A standard 2D image lacks absolute scale; a small object close to the lens can project onto the exact same pixels as a massive object far away.
Because of this inherent ambiguity, monocular depth estimation models do not predict absolute metric depth. Instead, they output relative depth maps that are accurate up to an arbitrary scale factor ($s$) and shift offset ($t$).
$$\text{Depth}_{\text{predicted}} = s \cdot \text{Depth}_{\text{true}} + t$$
When a neural network processes a video frame-by-frame, it optimizes for each image in isolation. Because it lacks context about the surrounding frames, the model picks a slightly different scale ($s$) and shift ($t$) for every single frame.
Yoon et al. point out that this view-variance causes severe jittering and ghosting artifacts when synthesizing new viewpoints over time. Even if the relative layout within a single frame is flawless, the global 3D geometry "breathes" and warps from frame to frame, causing surfaces to drift.
Strategy 1: Global Optimization and Post-Hoc Alignment
One way to solve this problem is to take independently predicted depth maps and align them so their scales and shifts match across time.
Early foundational work on bundle optimization demonstrated that optimizing depth maps simultaneously across large temporal windows (20 to 30 frames) using color similarity and geometric coherence drastically reduces flickering. Modern approaches build on this intuition but focus on efficiency. For instance, Artemyev et al. introduced a test-time optimization method that adjusts only local scaling coefficients across views rather than retraining entire networks, achieving temporal consistency at a fraction of the computational cost.
To handle more complex camera trajectories where rigid scaling fails, Huang proposed a flexible "deformation-spline" model. This approach allows for low-frequency geometric adjustments, effectively mitigating drift and flickering even on shaky, handheld smartphone footage.
[Frame t] --> [Per-Frame Model] --> Depth (Scale s_1) \
--> [Deformation Spline / Alignment] --> Stable Video Depth
[Frame t+1] --> [Per-Frame Model] --> Depth (Scale s_2) /
Similarly, Zhao et al. utilized locally weighted linear regression with a handful of sparse anchor points to mathematically bind inconsistent relative depth predictions into a stable metric trajectory.
Strategy 2: Spatio-Temporal Learning and Recurrent Networks
Instead of fixing depth maps after they are generated, a second paradigm integrates temporal constraints directly into the network architecture. By giving the model a "memory" of previous frames, it learns to enforce consistency natively.
Düzçeker et al. achieved this by integrating a convolutional LSTM (ConvLSTM) into a multi-view stereo pipeline. Crucially, their network accounts for camera motion by geometrically warping the hidden states over time, ensuring that features map to the correct physical locations from frame to frame.
Optical flow and multi-task learning also serve as powerful temporal anchors. Atapour-Abarghouei et al. enforced consistency by training a recurrent network to simultaneously predict depth and semantic labels, using optical flow loss to penalize temporal deviations.
Furthermore, looking both forward and backward in time prevents sudden depth flips. Bidirectional energy minimization strategies, as explored by Lin et al., ensure that complex visual challenges like occlusions and textureless regions (e.g., a blank wall or sky) do not cause the model's depth predictions to fluctuate erratically.
Why It Matters: Downstream Stability
Temporal consistency is not just about making videos look smoother; it is a strict requirement for downstream spatial computing tasks.
- Augmented Reality (AR): For a virtual object to look real, it must be realistically occluded by physical objects. As Watson et al. observe, per-frame depth instability causes the boundaries of virtual occlusions to jitter unpredictably, instantly breaking the illusion of presence.
- Generative Video Models: Modern AI video generators often rely on depth maps for structural guidance. Liu et al. demonstrated that if the underlying depth guidance suffers from scale drift or high-frequency flickering, the generative model inherits these flaws, producing unstable and noisy videos.
By stabilizing the underlying geometric signal—whether through plug-and-play stabilizers like the framework proposed by Shi et al. or joint spatiotemporal priors—the field is moving away from disjointed frame-by-frame inferences toward true, globally coherent 3D video understanding.