When a phone video of a moving person or pet is turned into a 3D scene that can be viewed from new angles, the reconstruction depends on a monocular depth network. This is a model that guesses how far away each pixel is from a single image. These guesses are useful but flawed in a specific way. Each frame's depth map is roughly right up to an unknown stretch and offset, and that stretch changes from frame to frame. Current systems respond in one of two ways. Some trust the depth, and its frame-to-frame errors flow straight into the 3D model. Others keep only the front-to-back ordering of pixels, which throws away the scene's real-world size. We propose a third option. Every frame contains static background, and the static background must have the same 3D shape in every frame. That makes it a free calibration target: it can measure each frame's stretch and offset and remove them. The dynamic foreground then receives depth supervision that is consistent over time and expressed in meters.
The opportunity matters because the hardest and most common capture, a casual handheld or tripod video with little camera motion, is exactly where reconstruction methods fail. Monocular Dynamic View Synthesis: A Reality Check shows that methods lose a further 4–5 dB of masked PSNR and about 30% of keypoint-transfer accuracy (PCK-T) on true monocular iPhone captures. Monocular Dynamic Gaussian Splatting: Fast, Brittle, and Scene Complexity Rules shows that Gaussian methods degrade quickly as camera motion shrinks. With little camera motion, depth along the line of sight is almost unconstrained, so the depth prior is the only information available. Calibrating that prior, rather than trusting it or discarding it, is the lever.
Core Claim
Dynamic 3D Gaussian splatting represents a scene as many small colored, semi-transparent ellipsoids that are rasterized very fast. Many such methods already split the scene into a static set of Gaussians and a moving set. We claim this split can be reused as a measuring instrument. The depth prior's error is modeled as a low-dimensional per-frame correction plus a per-pixel residual. The per-frame correction is estimated jointly with the reconstruction, using only pixels the model currently believes are static. Concretely, let $D_t(p)$ be the prior's depth at pixel $p$ in frame $t$. Let $\hat D_t(p)$ be the depth rendered from the static Gaussians, which are time-invariant by construction. Let $m_t(p) \in [0,1]$ be the model's current belief that pixel $p$ is static. We fit a per-frame log-scale $a_t$ and shift $b_t$ by minimizing
$$ \mathcal{L}_{cal} = \sum_t \sum_p m_t(p)\, \rho\!\left(e^{a_t} D_t(p) + b_t - \hat D_t(p)\right) + \lambda \sum_t (a_{t+1}-a_t)^2 , $$
Here $\rho$ is a robust penalty (Huber), and $\lambda$ weights a smoothness prior on scale changes between neighboring frames. Global metric scale is fixed once, as the robust median over frames of a metric depth model's prediction on static pixels. We use Metric3D v2, which produces metric depth from a single image given the focal length. Its frame-to-frame flicker averages out over the clip. The calibrated depth $e^{a_t}D_t + b_t$ then supervises the dynamic Gaussians. The loss is weighted mainly along the viewing ray, because that is the direction the photometric loss cannot constrain. Rendering speed does not change, since all of this machinery runs only during training.
Relation to Prior Work
Three systems are close neighbors, and each makes a different choice at the point where we intervene.
MoDGS reconstructs casual monocular videos with a canonical set of Gaussians and an invertible deformation network. It reports that direct metric depth supervision makes optimization fail, so it falls back on an ordinal loss that keeps only depth order. This result motivates our design: the failure comes from uncorrected per-frame drift, not from depth supervision as such. We keep magnitudes and remove the drift, where MoDGS gives up magnitudes altogether.
USplat4D also starts from the observation that monocular uncertainty is much larger along the line of sight than in the image plane. It acts on that observation by estimating per-Gaussian uncertainty and tying uncertain Gaussians' motion to confident "key node" anchors. USplat4D regularizes motion. We correct the upstream depth signal. USplat4D also never recovers metric scale, and it depends on 2D tracking succeeding. Its line-of-sight insight constrains our loss weighting, and it is our strongest baseline.
SDD-4DGS supplies the method for the static/dynamic split. It learns a per-Gaussian dynamic coefficient on a schedule that delays separation until geometry stabilizes, using DINOv2-based motion masks. It was evaluated on multi-camera and synthetic data with known scale. We reuse its split and give it a second job as the source of $m_t$.
WildGS-SLAM learns per-pixel uncertainty to keep moving objects out of a static map. It discards dynamics rather than reconstructing them. We use the same kind of static belief to calibrate the foreground instead of deleting it.
Two older lines supply the calibration idea itself. Fast-Depth / VIO alignment aligns relative depth to sparse visual-inertial points by least squares. LiDARTouch anchors depth with a few LiDAR beams. Both need an external metric source. Our anchor is internal: the time-invariance of the background, which exists even when the camera barely moves and there is no baseline for triangulation.
Research Program
We build on a dynamic Gaussian codebase that uses a shared deformation network. The benchmark study found that shared fields outperform per-Gaussian motion for monocular input (Monocular Dynamic Gaussian Splatting). To this we add the SDD-style static coefficient and the calibration loss above.
Training proceeds in three phases. The first is a short warm-up with the ordinal loss only, as in MoDGS, so that geometry and the static belief form. The second turns on joint estimation of $(a_t, b_t)$ from static pixels. The third supervises dynamic Gaussians with calibrated depth, using a loss that is stiff along the camera ray and loose across it.
Joint estimation matters because of the cascade. The static mask comes from 2D cues and is wrong at motion boundaries. A wrong mask biases the calibration, and a biased calibration distorts the geometry the mask is re-estimated from. To break this loop, $m_t$ is updated from the rendering residual of the current model, not frozen from a 2D segmenter. We will measure whether this loop converges or locks in early errors.
We also test a secondary hypothesis. The benchmark study's "catastrophic pruning" failure, where under-constrained Gaussians are deleted early and never recovered, should become rarer once Gaussians along the line of sight receive calibrated depth. We will log the number of Gaussians pruned per iteration and the number of holes that remain at the end.
Baselines
The naive reference point is the same codebase with no depth supervision. Next to it we run the ordinal-only variant, which reproduces MoDGS's choice. These two show how much any depth signal helps and how much is lost by discarding magnitudes.
The strongest existing method is USplat4D. On DyCheck at 2x resolution it reports a PSNR of 19.63, against 19.32 for the previous best, MoSca (USplat4D). We will run its released plug-in on the same base model, with the same depth and tracking priors and the same iteration budget. Beating it means exceeding 19.63 masked PSNR on DyCheck under its protocol, matching or improving PCK-T, and, most importantly, achieving lower metric depth error against iPhone LiDAR without per-sequence scale alignment. USplat4D does not claim to address that last quantity. No reported result was found on Lacuna for metric depth error of USplat4D or MoDGS on the iPhone LiDAR depth. We will therefore measure all methods ourselves with the same depth protocol.
MoDGS is run with its released settings as the second strongest competitor. Its reported DyNeRF PSNR is 22.64, against 19.55 for the next best baseline (MoDGS). We report on that split too, for continuity.
The ablation that isolates the key ingredient is a two-step pipeline. It calibrates each frame's depth by least squares against a temporal-median static depth computed from a fixed 2D motion mask, then trains the same model with that pre-calibrated depth. The only difference from our method is joint estimation with a reconstruction-driven static mask. Further ablations remove the smoothness term on $a_t$, replace the per-frame affine correction with a single global scale, and replace the line-of-sight weighting with an isotropic one. All variants share hyperparameter search budgets, the same depth model, and the same GPU-hour cap.
Why not just…?
Why not just supervise with Metric3D v2 metric depth directly? MoDGS reports that direct metric depth supervision makes optimization fail on casual video (MoDGS). The Metric3D v2 paper itself notes that its per-frame outputs flicker when applied to video (Metric3D v2). We still include direct supervision as a baseline. If it matches our method on iPhone data, the flicker is smaller than claimed and the calibration machinery is unnecessary.
Why not just align each frame's depth to the static background as preprocessing? This is the real threat, and it is the two-step ablation above. With clean motion masks it may do as well as our method. We expect it to fall behind where 2D masks are wrong: shadows, thin limbs, and objects that start moving partway through. In those regions the fixed mask biases $(a_t,b_t)$ and the bias is never revisited. We will stratify results by mask error measured against ground-truth co-visibility masks. If the two-step pipeline ties our method even in the high-mask-error stratum, we will report the contribution as the calibration principle, not the joint optimization.
Why not just use USplat4D's uncertainty weighting? USplat4D down-weights and anchors uncertain motion but leaves the depth signal uncalibrated and affine-ambiguous. It also cannot form anchors when tracking fails. Our method could even be combined with it. Running USplat4D with and without our calibrated depth will show whether the two gains add up or overlap.
Why not just use a simple time-conditioned field, which the Reality Check found competitive? That paper found a plain time-conditioned NeRF often beats deformation models on true monocular data (Reality Check). Such a model renders far slower than Gaussian splatting, which reaches up to 200 FPS (Monocular Dynamic Gaussian Splatting). It also provides no metric geometry. We report it as a quality reference, not as a substitute.
Proposed Experiments
The main benchmark is the DyCheck iPhone dataset (Reality Check). We report masked PSNR, SSIM and LPIPS, PCK-T, and metric depth error (absolute relative error and RMSE in meters) against its LiDAR depth, computed without any scale fitting. We stratify all metrics by the paper's effective multi-view factor, which measures how much the camera moves relative to the scene. Our prediction is that the gain grows as that factor shrinks.
A controlled sweep uses the SlidingCube and RotatingCube scenes (Monocular Dynamic Gaussian Splatting). It varies camera baseline at fixed object speed and locates where each method breaks down. We also report on DyNeRF and the MCV casual videos used by MoDGS, and we record training time and rendering FPS to confirm that rendering speed is unchanged.
Three checks would decide the question. First, metric depth error on iPhone must fall clearly below both the ordinal baseline and USplat4D. Second, our method must beat the two-step ablation in the high-mask-error stratum. Third, the measured pruning-collapse rate must drop in the low-camera-motion sweep. If the first check fails, the calibration does not remove drift. If the second fails, joint estimation is unnecessary. If only the third fails, the method still stands as a depth-calibration result but does not explain the brittleness that the benchmark study reported.
References
- Monocular Dynamic View Synthesis: A Reality Check
- Monocular Dynamic Gaussian Splatting: Fast, Brittle, and Scene Complexity Rules
- Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation
- MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos with Depth Priors
- Uncertainty Matters in Dynamic Gaussian Splatting for Monocular 4D Reconstruction
- SDD-4DGS: Static-Dynamic Aware Decoupling in Gaussian Splatting for 4D Scene Reconstruction
- WildGS-SLAM: Monocular Gaussian Splatting SLAM in Dynamic Environments
- Monocular Visual-Inertial Depth Estimation
- LiDARTouch: Monocular metric depth estimation with a few-beam LiDAR