Keeping Wires and Hair Steady in Video Depth

Models that estimate depth from a single camera are now good enough to run frame by frame on video. Their outputs flicker, though, so recent video depth models add a temporal training penalty that punishes depth for changing over time at each pixel. That penalty clashes with a second long-standing failure: thin structures such as cables, fences, branches and hair strands are already the first things a depth model loses. When a wire sweeps across a pixel, that pixel's true depth jumps from near to far within one frame. A loss that penalizes per-pixel temporal change reads this real jump as flicker and teaches the network to erase the wire.

We propose to change where temporal consistency is measured. Frame-wide jitter in depth scale and offset, which is what makes video depth "breathe," should be suppressed everywhere. Thin structures, by contrast, should be held consistent along their own motion, following the same physical point of the wire from frame to frame, rather than at a fixed pixel. Correspondences from synthetic renderers supply this motion during training only, so the model needs no optical flow at test time. We pair this loss with a small full-resolution correction branch that restores the thin geometry the patch-based encoder discards. We expect wires and hair that are both sharp and stable in video depth, measured on a new thin-structure video benchmark.

Core Claim

Current temporal objectives conflict with thin structures, and simple post-hoc fixes cannot remove the conflict. Video Depth Anything (VDA) adds a temporal attention head to a frozen Depth Anything V2 encoder. It trains with a Temporal Gradient Matching loss that compares predicted and true depth change over time at the same pixel. Exploiting Temporal Consistency for Real-Time Video Depth Estimation goes further and uses a 3D-convolutional discriminator to punish fast fluctuations outright. Both act at fixed pixel locations, so both treat a moving filament as noise.

Our claim is that the temporal loss should be split three ways. A frame-wide affine term controls scale and shift drift. The existing per-pixel gradient term is kept on bulky regions. On a thin-structure mask, a correspondence-anchored term replaces it. With $D_t$ the predicted depth at frame $t$, $D^{*}_t$ the ground truth, $m$ a thin-structure mask, and $\phi$ the ground-truth correspondence that maps pixel $p$ at time $t$ to the same surface point at $t-1$, the anchored term is

$$\mathcal{L}_{\text{anchor}} = \sum_{p:\,m_p=1} \left| \big(D_t(p) - D_{t-1}(\phi(p))\big) - \big(D^{*}_t(p) - D^{*}_{t-1}(\phi(p))\big) \right|.$$

It asks the network to get the depth change of the wire itself right, not the depth change of whatever pixel the wire happens to occupy. Because $\phi$ is used only as supervision, the deployed model never computes optical flow. That matters because flow is exactly what breaks on sub-pixel filaments, as Neural Video Depth Stabilizer and Robust Consistent Video Depth Estimation both show.

Research Program

Architecture. We start from VDA with its encoder frozen and its spatio-temporal head intact. We add a gated residual branch at full input resolution, modeled on the Depth Fixer of Guardians of the Hair. That module detects soft-boundary pixels, where a strand's color blends with the background, and corrects depth only there. It leaves the base model untouched over more than 95% of the image and raises Edge Precision on AIM-500 from 19.90% to 34.56%. The paper evaluates it only on single images and stereo pairs, and it has no temporal memory.

Our branch reads the high-resolution frame and VDA's output depth. It predicts a gate and a residual, and its features use a short causal temporal attention window over the previous frames. The gate is what the anchored loss trains through. Tying the branch to the gate also makes its effect easy to isolate in ablations. The branch addresses a second problem the temporal loss cannot: VDA's 14-pixel patches discard sub-pixel filaments before the temporal head ever sees them, so no temporal module can recover them.

Losses. Training combines four terms: a per-frame depth loss, a frame-to-frame scale-and-shift consistency term like the one in Online Video Depth Anything, VDA's gradient-matching loss masked to $1-m$, and $\mathcal{L}_{\text{anchor}}$ on $m$.

Training data. Training needs thin structures, dense depth and correspondences together, so all of it is synthetic. We render Infinigen-style procedural scenes rich in branches and foliage, plus inserted cables and fences, with moving cameras. We also build composite videos in the style of HairGuard's data pipeline: foregrounds from AIM-500 and P3M-10K mattes are placed over camera trajectories from RealEstate10K and DL3DV-10K, which gives exact alpha and correspondence by construction. The mask $m$ comes from ground-truth alpha values strictly between 0 and 1, or from thin geometry tagged in the renderer.

Analysis. Before building anything, we will measure the conflict directly. On the synthetic videos, we compute what fraction of VDA's gradient-matching penalty on ground-truth depth falls on thin-structure pixels, relative to their share of the image. This tells us whether the loss is lopsided before we claim it matters.

Baselines

Naive reference. Per-frame Depth Anything V2, with no temporal modelling at all, run at the same resolution as everything else. This sets the floor for flicker and the reference for edge sharpness. Its AIM-500 Edge Precision of 19.90% is reported in Guardians of the Hair.

Strongest temporal model. Unmodified VDA is the strongest competitor on consistency, because it is the model we extend and it reports top spatial accuracy on four of five standard benchmarks. The file gives no per-dataset number for it: no reported result was found on Lacuna for VDA's absolute metrics. For a fair comparison, VDA will also be fine-tuned on our synthetic thin-structure data with its original losses, using the same steps, GPU budget and hyperparameter sweep as our model. Without that, any gain could simply come from the new data.

Strongest thin-structure pipeline. HairGuard's Depth Fixer applied per frame on top of VDA output. This is the strongest thin-structure competitor, because it pairs the best edge repair with the best temporal base. Its single-image number to match is 34.56% Edge Precision on AIM-500. We will retrain the Fixer on our composite data under the same budget if the released weights do not cover wires.

Thin-structure repair followed by stabilization. HairGuard per frame followed by NVDS+, a post-hoc stabilizer that uses sliding-window cross-attention over neighbouring depth maps. This tests whether off-the-shelf stabilization can keep repaired edges intact.

Ablations. Three ablations isolate the key ingredient. The first removes the anchored term but keeps the masking: the gradient-matching loss is simply switched off on thin pixels, with nothing replacing it. The second keeps the anchored term and removes the residual branch. The third keeps the branch and trains it with plain gradient matching everywhere.

What counts as beating them. Our model beats the competitors if it lowers thin-region temporal error relative to VDA-plus-Fixer, matches or exceeds that pipeline's per-frame Edge Precision, and stays within noise of VDA on standard benchmarks.

Why not just…?

Why not just run VDA and then HairGuard's Depth Fixer on every frame? This is the real threat, and it is the main baseline. Its weakness is predicted rather than proven. The Fixer's gate and residual are computed independently per frame, so nothing ties the corrected wire depth at frame $t$ to frame $t-1$. Small gate changes should produce edge flicker exactly where VDA was stable. The thin-region flicker metric below settles this. If VDA-plus-Fixer matches us on it, the anchored loss adds nothing, and we would report that honestly.

Why not just switch off the temporal loss on thin pixels? That is the first ablation. Removing a penalty does not supply a signal: thin pixels would be left with no temporal constraint at all and would revert toward per-frame behaviour. Online Video Depth Anything already reports trailing artifacts on fast thin objects when temporal context is used without a structure-specific signal. If masking alone matches the full method, we will adopt the simpler recipe.

Why not just stabilize afterwards with NVDS+ or deformation splines? NVDS+ falls back to raw per-frame depth wherever optical flow is large or uncertain, which is precisely the condition on moving filaments. Kopf et al. report residual wobble because their spline grid is too coarse for sub-pixel detail. Both pipelines appear as baselines, so the claim is tested rather than assumed.

Why not just run at higher resolution? Temporal attention cost grows sharply with the number of tokens, and the mixed foreground-background pixels at strand edges exist at any resolution. We include VDA run at twice the resolution, with its latency reported, as a compute-matched comparison.

Why not use Vid2Curve? Vid2Curve reconstructs only isolated networks of 3D curves and needs clean segmentation masks. It produces no dense depth for walls or ground. CurveFusion has the same limitations and additionally needs a depth sensor. Both are reference points for wire-only scenes, not for full video depth.

Proposed Experiments

Thin-structure video benchmark. We will hold out synthetic videos with wires, branches and composited hair. On these we report four things:

  • per-frame Edge Precision;
  • depth error restricted to thin-structure pixels;
  • thin-region temporal error, defined as the anchored residual above evaluated with ground-truth correspondences;
  • a standard pixel-level flicker measure such as TCC.

The fourth is included to show where pixel-level metrics misjudge moving filaments.

Real footage. On Marvel-10K stereo video, we will measure left-right reprojection consistency of the predicted depth on hair regions across frames. HairGuard reports gains there in PSNR and SIoU, but no numbers were found on Lacuna. On AIM-500 we will compare single-frame Edge Precision against the reported 34.56%.

No regression elsewhere. We will run KITTI, Sintel, Bonn and NYUv2 under VDA's protocols to confirm that bulky-scene accuracy and long-video drift are unchanged. For reference, STATIC reports KITTI RMSE 1.977 and NYUv2 δ<1.25 of 0.934.

Decisive checks. Three outcomes would decide the question:

  • the full model beats VDA-plus-Fixer on thin-region temporal error at equal or better Edge Precision;
  • the masking-only ablation falls between the two, which shows the anchored signal, and not just removing the penalty, does the work;
  • the gain grows with filament speed when results are binned by ground-truth image-space velocity, which is where pixel-level losses should fail most.

If none of these hold, the conflict between temporal smoothing and thin structures is smaller than the literature suggests, and that is itself worth reporting.