Videos That Carry Their Own Metric Geometry

Camera-controlled video generators are image-to-video models that take a camera path as input and render the scene as if filmed along that path. They have become good at following camera paths in direction and shape. They are still bad at following them in distance. The reason is monocular scale ambiguity: a single image cannot tell a small room from a large one, because scaling a scene uniformly leaves its picture unchanged. Training trajectories estimated by structure-from-motion therefore come with an arbitrary scale per clip, and a command like "move forward one meter" means something different in every scene. Within a single long generated rollout, the implied size of the world can also drift. Today's geometric regularizers do not see this failure. They check that frames agree with each other relative to the generator's own depth, not that the depth and motion agree with the physical world.

We propose that the generator output, alongside every RGB frame, a dense 2D-to-3D correspondence map. This is a per-pixel "scene coordinate": the 3D location, in meters, of the surface each pixel sees, in one fixed world frame anchored at the first input image. Because every frame points into the same metric frame, a standard pose solver can read off the camera the generator actually rendered. That solver is Perspective-n-Point with RANSAC: it recovers camera pose from 2D–3D matches and rejects matches that disagree. The generator's recovered camera can then be compared against the commanded one, and scale is fixed once at the first frame instead of being re-guessed every frame. Pixels on moving objects fall out naturally as solver outliers, so rigidity is enforced where the world is rigid without rewarding frozen motion.

Core Claim

A video generator that must produce metric scene coordinates in a single shared frame will follow commanded camera translation magnitudes and hold a consistent world scale over long rollouts better than one that only receives camera rays. It will also do this without the "static collapse" that rigid consistency penalties cause. The new ingredient is not metric conditioning alone, which a simple rescaling of training poses can provide, as we discuss below. It is making the generator's output geometrically self-verifying: the camera it rendered can be recovered from its own pixels, with no external reconstruction model acting as judge.

Research Program

Prior work falls into three groups. The first supplies the mechanism. Dense coordinate regression followed by a robust pose solver is the standard recipe in object pose estimation, as in Pix2Pose, which regresses per-pixel object coordinates as an image-to-image task. The same recipe appears with per-pixel uncertainty weighting RANSAC, as in Uncertainty-Driven 6D Pose Estimation. It also appears in scene-scale relocalization, as in Scene Coordinate Reconstruction Priors. We reuse this recipe but place the coordinate predictor inside a generator rather than a recognizer.

The second group motivates the gap. CamCo conditions Stable Video Diffusion on per-pixel Plücker rays, which encode each pixel's viewing line, and restricts attention to epipolar lines. It assumes fixed intrinsics and never anchors scale. CameraNoise warps the initial noise using a constant, arbitrary pseudo-depth. GeoVideo generates depth jointly with RGB and penalizes reprojection disagreement, but its depth has no enforced physical scale and its pose and depth pseudo-labels come from upstream models. CamGeo distills geometry from a VGGT teacher in normalized coordinates.

The third group constrains the design. Stream4D shows that rigid consistency rewards teach generators to stop moving. Any loss we add must therefore not grow when more pixels become static.

We build the model on Stable Video Diffusion with CamCo-style Plücker conditioning. We add three coordinate channels decoded from the shared latent, in the same way GeoVideo adds depth.

Training labels come from RealEstate10K and DL3DV-10K. We run Metric3D v2 on each frame with the dataset's known intrinsics. That model addresses the confusion between focal length and scale by mapping images into a canonical camera, and it reports zero-shot Abs Rel of 0.044 on KITTI and 0.047 on NYUv2. We then fit a single scale factor per clip between the structure-from-motion sparse points and the metric depth, using the closed-form least-squares anchoring that LiDARTouch-style work (Zablocki) and Wofk et al. use with sparse sensors. Fitting one scale across the whole clip, rather than one per frame, cancels much of the per-frame jitter of the depth model. Finally, we transform per-frame depth into first-frame world coordinates using the rescaled poses.

The geometric training signal is computed on the generator's own predicted coordinates. For each generated frame $t$ we solve PnP with RANSAC on its predicted coordinate map, obtaining a recovered pose $\hat{T}_t$ and an inlier set $I_t$. The loss is

$$ L_{\text{self}} = \frac{1}{|I_t|}\sum_{p \in I_t} \big\| \pi(T_t^{\text{cmd}}, X_t(p)) - p \big\| $$

Here $T_t^{\text{cmd}}$ is the commanded camera, $X_t(p)$ is the predicted metric coordinate at pixel $p$, and $\pi$ is perspective projection. The loss measures, over the rigid inliers only, how far the commanded camera is from the geometry the generator produced.

We average over inliers and never reward the inlier count, so freezing a moving object does not lower the loss. The Stream4D constraint therefore holds by construction, and we test it rather than assume it. Moving pixels are left to the ordinary diffusion loss. Their coordinates are free to change over time, which is the non-rigid regime where Jamais Vu finds that rigid alignment breaks down. We do not attempt to solve articulated correspondence here.

Baselines

The naive reference point is CamCo as published: Plücker conditioning on per-clip-normalized structure-from-motion poses. It reports FID 14.66, FVD 138.01 and a COLMAP reconstruction error rate of 3.8% on RealEstate10K, and FID 22.19 with FVD 137.59 on its dynamic WebVid subset. We retrain it on our exact data mix, resolution and step budget rather than rely on those numbers. Matching them serves as a sanity check.

The strongest competitor on the scale question is a baseline we construct: metric-rescaled CamCo, which is CamCo trained on the same metric-rescaled trajectories as our model but with no coordinate output. It isolates whether metric conditioning alone explains any gain. It is the most dangerous baseline, and beating it is the bar for the core claim. Beating it means lower metric translation-magnitude error and lower long-horizon scale drift at equal FVD, not better FID alone.

The strongest existing geometric regularizer is GeoVideo-style RGB-plus-depth generation with a motion mask and reprojection loss. We port it to the same Stable Video Diffusion backbone and give it the same metric depth labels, so the only difference is depth-with-commanded-pose versus coordinates-with-recovered-pose. GeoVideo reports a multi-view consistency score improving from 61.2 to 77.2 on DL3DV. That result is on CogVideoX-5B, so our port is not directly comparable.

We also include CamGeo's VGGT-distillation objective as the teacher-based alternative. It reports roughly 16% lower rotation error at quarter-sparse RealEstate10K keyframes.

Two ablations isolate the key ingredient. The first keeps the coordinate channel but drops $L_{\text{self}}$, testing whether predicting coordinates alone helps. The second replaces RANSAC inliers with GeoVideo's learned motion mask. All models share compute, data, tuning trials, and seeds.

Why not just…?

Why not just rescale the training poses to meters and keep Plücker conditioning? This is a real threat. It may fully fix scale on the first frames, since the conditioning then carries metric meaning. What it cannot do is detect when the generator's rendered world drifts in scale during a long rollout, because nothing in its output exposes the camera it actually rendered. The metric-rescaled CamCo baseline settles this. If it matches us on both short-clip translation error and long-rollout drift, the coordinate channel has no value for scale, and we would reposition the work as a self-verification tool only.

Why not just predict depth, as GeoVideo does, and reproject with the commanded pose? Depth combined with the commanded pose presupposes that the generator obeyed the command, so the loss penalizes appearance for disagreeing with a camera that may never have been rendered. Recovering the pose by PnP separates "wrong geometry" from "wrong camera". The depth-channel port on the same backbone tests whether this separation matters.

Why not just score geometry with an external reconstructor like VGGT or a 4D Gaussian model? Teacher errors propagate into the generator, as GeoVideo and ReconX acknowledge. Stream4D also notes the risk of alignment bias when the same family of reconstructors both rewards and evaluates the generator. Our model still inherits Metric3D bias through its labels. We therefore evaluate with an independent protocol, described below, that compares against held-out real frames rather than against any learned model.

Why not a rigid 3D Gaussian splatting reward? Stream4D documents that it produces near-stationary video. We include motion scores specifically to show that our inlier-averaged loss does not reproduce this failure.

Proposed Experiments

The first decisive check uses real held-out frames. On RealEstate10K and DL3DV test clips, we condition on the first frame and the ground-truth trajectory in meters, obtained by the same sparse-point rescaling. We then measure PSNR and LPIPS against the true later frames. A generator with the wrong scale produces the wrong parallax and loses pixel agreement, so this metric punishes scale errors without relying on any learned judge.

The second check sweeps scale. We command the same path at 0.5, 1 and 2 times its metric length and measure whether the parallax, taken as optical-flow magnitude on static regions, scales proportionally. A model that ignores metric magnitude shows a flat response.

The third check measures long-horizon drift. On 100-frame autoregressive rollouts, we track the ratio of the translation recovered by COLMAP-then-rescaled against the commanded translation over time. We apply this protocol to all methods, including ours, rather than using our own PnP readout, so the evaluation is not graded by our loss.

The fourth check tests motion preservation. On the WebVid dynamic subset and VidProM motion-prominent prompts, we report dynamic degree and a motion score in the 0.7–0.8 range that Stream4D reports. The claim fails if our model's motion drops measurably below metric-rescaled CamCo's.

Finally, we report the standard FID, FVD, COLMAP error rate, and rotation and translation errors on RealEstate10K against CamCo's 3.8% error rate. The proposal succeeds only if the coordinate model beats metric-rescaled CamCo on the first and third checks at comparable FVD and motion. Winning only on the second check would suggest the gain comes from metric labels, not from self-verification.