Plausibility-Gated Temporal Updates for Synthetic-to-Real Video Depth

Video models pretrained on large amounts of real footage have absorbed a great deal about how the world moves. When such a model is fine-tuned on video-game clips to predict depth, the game supplies perfect depth labels, but its motion can be subtly wrong: scripted camera paths, stuttering frame timing, simplified physics. Current practice picks one of two blunt options. One is to update everything, which risks overwriting the real-world motion knowledge with game-engine motion. The other is to freeze the motion-handling layers entirely, which means they can never adapt to what depth prediction actually needs. We propose a third option. The pretrained model itself scores each synthetic clip on how plausible its motion looks. Every clip teaches the layers that handle single-frame appearance. Only clips whose motion the model finds plausible are allowed to update the layers that handle time, and the plausibility bar is lowered gradually over training.

In plain terms, the model's own motion prior decides which synthetic clips it may learn time from, while all clips can still teach it appearance and geometry. This is a curriculum, an ordering of training from easy to hard. Here, though, the ordering acts on which parameters receive gradients from which data, not on which data is shown. The difficulty measure also comes from the model, not from a hand-picked heuristic.

Core Claim

When a real-video-pretrained model is adapted to synthetic data with exact labels, the domain gap has two parts: appearance (rendering) and dynamics (simulated motion). These call for different treatment. Appearance errors are what the fine-tuning should correct. Dynamics errors are what the pretrained temporal layers should be protected from. We claim that gating temporal-layer gradients per clip, using a motion-plausibility score computed from the frozen pretrained model and relaxed on a self-paced schedule, gives better temporal consistency on real benchmarks than either full joint fine-tuning or a static temporal freeze, at equal spatial accuracy. We further claim this advantage grows sharply when the synthetic data contains dynamics artifacts. The second claim can be tested cleanly by injecting such artifacts.

Where This Sits

The work that motivates the proposal shows that separating space from time matters under domain shift. ToonCrafter freezes the temporal attention of a live-action video diffusion model and trains only the spatial layers on cartoon clips. ToonComposer and AnyMoLe similarly restrict updates to spatial modules. SeqHAND trains on synthetic hand "pose-flows" and then freezes its recurrent temporal layer while adapting the image encoder to real data. All four use a binary, all-or-nothing freeze fixed before training. We keep their intuition but make the temporal freeze depend on the clip and on the training stage.

The work that constrains the proposal is Depth Any Video. It fine-tunes Stable Video Diffusion jointly on 6 million game frames (the DA-V dataset) with no decoupling and no curriculum, and still reports the lowest Temporal Alignment Error (TAE), 2.1, on ScanNet++. This is the null hypothesis we must beat. Joint fine-tuning may simply not forget much when game motion is mostly realistic camera motion. We address it directly below. The same paper also reports persistent failures on water reflections, a reminder that some of the gap is optical, and our method does not claim to fix that.

The work that supplies the curriculum idea is Diffusion Curriculum. It builds a synthetic-to-real continuum of images by varying diffusion guidance, and raised ImageNet-LT tail accuracy from 4.4% to 23.64% by moving along it. It works only on static images and updates all parameters uniformly. We borrow its "clean first, harder later" pacing and the self-paced thresholding of Self-Paced Learning for Latent Variable Models, which admits samples by a relaxing difficulty threshold. We apply both to temporal gradients instead of to data inclusion. Video Depth Anything freezes a spatial image encoder and trains a temporal head with a temporal-gradient-matching loss. That is the opposite freeze direction: its temporal prior is learned from scratch, so there is no pretrained motion knowledge to protect. We treat it as a comparison point for what temporal consistency costs, not as a competitor in the same regime.

Research Program

The plausibility score. We need a per-clip measure of whether the pretrained temporal layers "recognize" a clip's motion, and it must not be confounded by unfamiliar game rendering. Let $L_{\text{full}}(x)$ be the frozen pretrained video model's denoising loss on RGB clip $x$, averaged over a few fixed noise levels. Let $L_{\text{sp}}(x)$ be the same loss with temporal attention layers bypassed, so each frame is processed independently. We define

$$ s(x) = L_{\text{sp}}(x) - L_{\text{full}}(x), $$

which is the amount the pretrained temporal layers help predict this clip. Both terms see identical frames, so rendering unfamiliarity largely cancels. The prediction is that a high $s$ means the motion matches the prior, and a low or negative $s$ means the temporal prior is misled. The score is computed once, offline, with two forward passes per clip at a handful of noise levels, which is affordable for 40,000 clips.

Gated training. Fine-tuning follows the Depth Any Video recipe: flow matching toward depth latents, starting from Stable Video Diffusion. We change one thing. For each clip, spatial parameters always receive the depth loss gradient. Temporal parameters receive it only if $s(x) \ge \tau_t$, where the threshold $\tau_t$ at training step $t$ starts at a high quantile of the score distribution and decreases linearly to a final quantile $q$, chosen on a held-out synthetic split. Clips below the threshold update only the spatial parameters.

What gets analyzed. Beyond benchmark numbers, we track two diagnostics. The first is how far the temporal parameters drift from initialization under each method. The second is whether $s$ recovers known dynamics corruptions. For that check we take clean DA-V or Virtual KITTI 2 clips and create corrupted copies by reversing time, dropping frames irregularly, or repeating frames to mimic stutter. A useful score must rank these corrupted copies below their clean originals. If it does not, the method has no basis and we stop.

Baselines

The naive reference point is full joint fine-tuning with a uniform learning rate: the Depth Any Video recipe, reproduced at reduced scale. It is also the strongest existing method on this exact question, since it reports TAE 2.1 on ScanNet++, the lowest among the models it compares. Our compute will not match the original run. So every arm is trained from the same Stable Video Diffusion checkpoint, on the same DA-V subset, for the same number of steps, with the same learning-rate sweep and three seeds. Beating it means a lower TAE than our reproduction on ScanNet++ and the other real sequences, with AbsRel and δ1 no worse, and with the effect consistent across seeds. The published 2.1 is reported alongside as an external anchor. Our reproduction's value relative to it tells readers how much the reduced scale costs.

The static temporal freeze follows ToonCrafter and SeqHAND: temporal layers are never updated. It is the strongest competitor for the "protect the prior" half of the claim. It runs with the same budget and tuning. Beating it requires better temporal consistency on real data at equal spatial accuracy. If we cannot beat it, the evidence says that temporal layers need no depth-specific adaptation at all.

Two ablations isolate the key ingredient, which is per-clip selection by $s$. The first is random gating: temporal gradients are allowed on a randomly chosen subset of clips whose size follows the same schedule as the real gating. This separates "which clips" from "how many temporal updates." The second is inverted gating, which updates temporal layers only on low-score clips. It should be clearly worse if the score means anything. For context, we also report Video Depth Anything's released model where it can be run on the same sequences. The file reports no per-benchmark numbers for it: no reported result found on Lacuna for Video Depth Anything's individual AbsRel or TAE values.

Why not just…?

Why not just freeze the temporal layers, as ToonCrafter and SeqHAND do? This is the most serious threat. If game motion is harmful, freezing already protects against it. If it is harmless, freezing costs little. The case against a static freeze is that predicting depth, rather than RGB, may need different temporal behavior: stable depth under lighting flicker, for example, which the RGB prior never had to learn. Whether this matters is an empirical question, and the static-freeze arm answers it. If static freezing matches gating on clean data, we would still expect gating to win only if some temporal adaptation helps. We will report this outcome plainly rather than argue it away.

Why not just unfreeze temporal layers gradually, or give them a smaller learning rate? Scheduled unfreezing controls when temporal layers learn but not from what. After unfreezing, they learn from bad-motion clips as readily as from good ones. The random-gating ablation, together with a scheduled-unfreezing arm on the same timetable, tests exactly this. The corrupted-dynamics stress test is where the difference should be largest: a schedule has no way to avoid stuttered or reversed clips, and the score does.

Why not just filter implausible synthetic clips out of training? Filtering throws away the clips' exact depth labels, which still carry good per-frame geometry. We will run a filter arm that drops low-score clips entirely. Gating should match or beat it on spatial accuracy (AbsRel, δ1) because it keeps those labels for the spatial layers. If filtering is equally good on every metric, the simpler method wins, and we will say so.

Why not just make the synthetic data more realistic, or translate it toward real appearance as Diffusion Curriculum does? Virtual KITTI 2 shows that better rendering closed roughly half the sim-to-real gap, not all of it. Rendering and per-frame translation both act on appearance. Frame-wise diffusion translation can also introduce temporal inconsistency. Neither addresses flawed dynamics, which is the part of the gap our method targets. The two approaches are complementary, not competing.

Proposed Experiments and Decisive Checks

Training uses DA-V clips as described in Depth Any Video. No resource id is given for DA-V, so if it is not obtainable we substitute Virtual KITTI 2 plus a smaller self-rendered set and state this. Evaluation is zero-shot on ScanNet++, NYUv2, KITTI, ETH3D, Bonn and Sintel, using AbsRel and δ1 for spatial accuracy and TAE for temporal consistency. ScanNet++'s iPhone streams give a further real-sensor test with motion blur and rolling shutter. We also run a per-source analysis that splits GTA V clips from Cities: Skylines II clips, since their camera motion differs, and check whether the score distributions differ as expected.

Three checks decide the proposal. First, the score must rank corrupted-dynamics clips below their clean versions with high agreement; otherwise the method is abandoned. Second, in the stress test we mix 20% corrupted-dynamics clips into training. Joint fine-tuning should lose temporal consistency on real benchmarks. Gating should keep it close to the static-freeze level while preserving spatial accuracy. If gating does not beat joint fine-tuning here, the core claim fails. Third, on clean data, gating must beat both static freezing and random gating on TAE at equal AbsRel. If it only ties static freezing, we conclude that the right lesson for this setting is "freeze time," and the evidence for that conclusion is itself a useful result, given that the strongest published method does not freeze.