When a depth network runs on a video one frame at a time, the scene appears to "breathe." Each frame's depth is only correct up to an unknown scale and offset, and the network picks slightly different values every frame. Current fixes fall into two groups. Some smooth the depth over time, either with learned temporal layers or with alignment between overlapping windows. Others recover camera poses with structure-from-motion and fit the depth to them. The first group makes depth stable but not necessarily right. The second is accurate but fails exactly where pose estimation fails.
Light-field imaging offers a third option. In an epipolar plane image (EPI), which is a 2D slice that stacks the same pixel row across a line of viewpoints, every static point traces a straight line whose slope equals its disparity, or inverse depth. Points infinitely far away trace vertical lines with zero slope. We propose to rebuild this slice from ordinary video, using short point tracks and the two-view geometry between frames instead of a calibrated camera array. We then require that the slopes observed in the video agree with the inverse depth the network predicts. This agreement acts as a physical measurement, not a smoothness prior. It fixes the per-frame offset outright, ties per-frame scales together through overlapping frame pairs, and needs no global camera poses. It can serve as a self-supervised fine-tuning loss on unlabeled real video and as a closed-form correction at test time.
Core Claim
Temporal depth inconsistency is a mismatch between two EPIs: the one formed by image appearance and the one formed by predicted depth. Light-field methods enforce straightness within the image EPI. Light Field Spatial Super-Resolution via Deep Combinatorial Geometry Embedding and Structural Consistency Regularization uses an EPI gradient loss for this, and LFIC-DRASC uses elongated strip kernels along EPI lines. Video depth methods instead enforce smoothness within the depth slice. DVD's latent temporal-gradient supervision in DVD: Deterministic Video Depth Estimation with Generative Priors is an example. Neither line of work requires the two slices to agree with each other, and that agreement is the constraint that actually encodes geometry.
Take a static point tracked from frame $i$ to frame $j$. After removing the rotation implied by the two-view epipolar geometry, its displacement along the epipolar line obeys
$$\frac{p_{ij}(x)}{g_{ij}(x)} = \tau_{ij}\,\rho_i(x).$$
Here $p_{ij}(x)$ is the observed displacement in pixels, $g_{ij}(x)$ is a known geometric factor set by the pixel's position relative to the epipole, $\tau_{ij}$ is the unknown translation magnitude, and $\rho_i(x)$ is the true inverse depth. If the network outputs inverse depth that is correct only up to an affine map, a robust linear fit of $p/g$ against the prediction must have zero intercept and small residual. A nonzero intercept means vertical lines at infinity have been tilted, which is a shift error. A per-pair slope that is inconsistent across overlapping pairs means scale breathing. Points on moving objects do not fit the epipolar geometry, so a robust epipolar-geometry fit that discards outliers (RANSAC) flags them and removes them from the loss. This addresses the dynamic-scene failure that affects rigid reprojection methods.
Relation to Prior Work
Several papers motivate the gap. Consistent video depth estimation gets strong stability from reprojection losses, but it requires COLMAP poses and about 40 minutes of test-time fine-tuning per 244-frame clip. Medea shortens this to 0.4 s/frame by optimizing only a coarse spline of scale corrections, but it still assumes accurate poses and static scenes. Our constraint uses the same physics but needs only pairwise relative geometry from short tracks, which can be estimated over a few frames even when global SfM breaks.
Other papers supply the base models we build on. StableDPT adds temporal attention to a frozen DepthAnythingV2 encoder and attends to global keyframes. DVD regresses depth deterministically with a video diffusion transformer and aligns windows with a closed-form affine solver. Learning Temporally Consistent Video Depth from Video Diffusion Priors (ChronoDepth) stabilizes sliding windows with clean depth anchors. All three obtain consistency only from learned priors and from depth-to-depth agreement. DVD's affine coherence, for instance, aligns one window's prediction to another's, so a shared error propagates instead of being corrected. Our loss compares predicted depth to an independent geometric measurement.
Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks and DeepVideoMVS do use epipolar warping across frames. Both, however, require known poses at inference and build cost volumes, which fail on textureless surfaces. We use epipolar geometry only where tracks exist, as a training and correction signal, and let the learned prior fill textureless regions.
On the multi-view side, EpiDiff and DGE restrict cross-view attention to epipolar lines so that generated views agree. They show that epipolar locality is a strong and cheap consistency constraint, but only for static scenes with known cameras. We take from them the idea that epipolar structure, not dense 3D, is enough to enforce agreement across views. Multi-view consistency also defines how we evaluate: depth from different frames must describe one scene.
The file also contains a warning, from the light-field literature itself, that EPI lines are only straight for dense, uniform, rectilinear camera grids. Free-form video breaks this. We respond by never assuming straight lines in raw x–t slices. We test only the pairwise slope relation above, which holds for any translation. We also down-weight pairs with near-zero translation (pure rotation), where the relation carries no information.
Research Program
The work has four parts.
First, we build the track-rectified EPI extractor. It takes short point tracks from an off-the-shelf tracker or optical flow. For frame pairs within a window, it estimates the essential matrix robustly, removes rotation, and computes the ratio $p/g$ for each track, along with an inlier mask.
Second, we define the parallax-slope loss. For each pair, we fit the slope and intercept of $p/g$ against the predicted inverse depth in closed form. The loss penalizes the intercept, the inlier residual, and disagreement among the fitted slopes $\tau_{ij}$ across overlapping pairs, which a translation-consistency check across triplets ties together.
Third, we fine-tune a feed-forward model on unlabeled real video with this loss added to the model's original synthetic-data supervision. We choose a StableDPT-style model as the primary base because it trains on a single consumer GPU in 3 days, and we test transfer to DVD's backbone where compute allows. The unlabeled video comes from the VDW videos used by Neural Video Depth Stabilizer.
Fourth, we apply the same fit at inference as a closed-form per-frame affine correction, applied without any gradient steps, and compare it against DVD's window alignment.
Baselines
The naive reference is per-frame DepthAnythingV2 depth followed by sliding-window least-squares affine alignment. This is the procedure that MEDeA's single-view baseline reflects. It reached 14.3% AbsRel on TUM RGB-D in the MEDeA paper, against MEDeA-M's 6.4% with poses.
The strongest feed-forward competitor is DVD. It reports AbsRel 5.5 on ScanNet and 6.7 on KITTI, and boundary F1 0.259 against Video Depth Anything's 0.210 on ScanNet. I found no reported result on Lacuna for DVD's temporal flow-warping error, so we will measure it ourselves. StableDPT is the matched-architecture baseline, since our main model shares its backbone. We will report its released model and also a version fine-tuned on exactly the same unlabeled videos, for the same steps and on the same GPU budget, using a flow-warping smoothness loss instead of ours. This is the fair comparison.
For pose-based methods, we run MEDeA and COLMAP-pose reprojection alignment, the CVD-style approach, on identical clips. We count failures of the pose stage as failures of the method rather than dropping those clips.
The key ablation replaces the parallax-slope loss with a temporal gradient penalty on the depth slice, analogous to DVD's latent rectification. A second ablation substitutes ground-truth poses for the estimated pairwise geometry. A third removes outlier rejection.
Beating the field means three things. On ScanNet and KITTI, we must match DVD's per-frame-aligned AbsRel within noise. Under sequence-level alignment, we must clearly improve on it. On the pose-failure subset, we must beat the pose-based methods.
Why not just…?
Why not just add an optical-flow warping or temporal smoothness loss? Flow measures 2D motion and says nothing about scale or offset. A depth video can be perfectly smooth and still wrong by a slowly drifting scale. NVDS+ had to add flow-guided fusion specifically to suppress ghosting from direct flow warping in high-motion regions. Our smoothness-loss ablation, trained on the same data, settles this directly. If it matches our loss under sequence-level alignment, our central claim is false.
Why not just run COLMAP or SLAM and align depth to poses, as CVD and MEDeA do? This is the real threat. On well-textured, slow static clips it will probably match or beat us, because global poses carry more information than pairwise geometry. Our bet is on two other things: clips where global registration breaks, such as rapid motion, textureless walls, and dynamic scenes like Bonn and Sintel; and cost, since we use feed-forward inference with a closed-form correction instead of minutes of optimization. We will report results separately on clips where COLMAP succeeds and where it fails. If the pose-based methods dominate in both groups, the contribution narrows to speed.
Why not just use DVD's global affine coherence or StableDPT's keyframe attention? Both align predictions to other predictions, so any error the model makes consistently is never corrected. DVD itself notes that its affine alignment breaks under rapid ego-motion. Our test-time correction is a drop-in replacement for DVD's solver on DVD's own outputs, which isolates the measurement-versus-self-agreement question.
Proposed Experiments
We evaluate zero-shot on ScanNet, KITTI, TUM RGB-D, Sintel, and Bonn. Accuracy is AbsRel and δ<1.25 under two protocols. The first is the standard per-frame alignment. The second fits a single scale and shift to the whole sequence. The standard protocol hides breathing, so the sequence-level protocol is the decisive one. Temporal stability is measured by optical-flow warping error. Multi-view consistency is measured by reprojecting each frame's depth into frames 10, 30, and 60 apart using ground-truth poses on ScanNet and TUM RGB-D and reporting the depth disagreement. We also report runtime per frame.
Three checks decide the outcome. First, on ScanNet and TUM, the sequence-level AbsRel gap between our fine-tuned model and the smoothness-loss model must exceed the seed-to-seed variance. Second, the fitted intercept must be near zero after fine-tuning and large before. This is a direct readout showing that vertical lines at infinity have been restored, and we can visualize it in rectified EPI slices. Third, on dynamic sequences, performance must not degrade relative to the base model. If it does, the outlier mask is letting moving points corrupt the constraint, and the method's dynamic-scene claim does not hold.
References
- Light Field Spatial Super-Resolution via Deep Combinatorial Geometry Embedding and Structural Consistency Regularization
- LFIC-DRASC: Deep Light Field Image Compression Using Disentangled Representation and Asymmetrical Strip Convolution
- DVD: Deterministic Video Depth Estimation with Generative Priors
- Consistent video depth estimation
- MEDeA: Multi-view Efficient Depth Adjustment
- StableDPT: Temporal Stable Monocular Video Depth Estimation
- Learning Temporally Consistent Video Depth from Video Diffusion Priors
- Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks
- DeepVideoMVS: Multi-View Stereo on Video with Recurrent Spatio-Temporal Fusion
- EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion
- DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editing
- Neural Video Depth Stabilizer