Gravity as a Ruler: Metric, Non-Linear Frame Interpolation from Monocular Video

A single camera cannot tell a small nearby scene from a large distant one. Depth networks trained on single images learn which object is in front of which, but not how many meters away anything is. Video frame interpolation (synthesizing the missing frames between two captured ones) has a related blind spot. Most methods assume every pixel moves in a straight line at constant speed between frames, which breaks for thrown, falling and bouncing objects. The two problems share a remedy that has been overlooked. A thrown ball accelerates downward at 9.81 m/s² regardless of the camera. If we track a free-flying object in a video with a known frame rate, the size of its acceleration fixes the absolute scale of its 3D path. That scale then transfers to the rest of the scene.

We propose to turn this observation into a working system in two steps. First, it recovers metric scale, and where possible focal length, from ballistic segments of ordinary monocular video, with no depth sensor and no camera metadata. Second, it uses the resulting metric 3D trajectories to interpolate frames along physically correct curved paths from only two input frames. It fills the background uncovered along the true curved path from a depth-aligned background layer built from the rest of the clip. The payoff is concrete. Constant-velocity methods produce ghosting and smear artifacts around fast objects because they put the object in the wrong place. Gravity supplies the missing acceleration that two frames alone cannot provide.

Core Claim

A perspective camera cannot distinguish a trajectory from the same trajectory scaled up by any factor about the camera center. The unknowns are the starting point $X_0$, velocity $v$ and acceleration $g$, all in 3D. Multiplying all three by the same factor $\lambda$ leaves every projected image point unchanged. Monocular scale ambiguity is exactly this $\lambda$. For an object in free flight, however, $\|g\| = 9.81\,\mathrm{m/s^2}$ is known, and time is known from the frame rate. Fitting the projected model

$$x(t) = \pi\!\left(K\left(X_0 + v\,t + \tfrac{1}{2} g\,t^2\right)\right), \qquad \|g\| = 9.81$$

to tracked image positions $x(t)$ therefore pins down $\lambda$. Here $\pi$ divides by depth and $K$ holds the camera intrinsics. The same fit also recovers the gravity direction. When the object moves appreciably in depth, the perspective distortion of the arc also constrains the focal length inside $K$. A relative video-depth map aligned to the object's recovered metric depth then becomes metric everywhere.

The fitted trajectory also removes the core obstacle in two-frame interpolation. BiM-VFI identifies what it calls time-to-location ambiguity: from two frames, an accelerating object's position at intermediate time $t$ is undetermined, so trained networks blur toward the average. At test time BiM-VFI falls back to uniform velocity. Quadratic Video Interpolation resolves the ambiguity by estimating acceleration from four frames. In metric space with known $g$, the endpoints $X(0)$ and $X(1)$ determine $v$ uniquely, so the full curve follows from two frames.

This is our specific claim: gravity-known metric 3D interpolation beats both learned two-frame non-linear interpolation and four-frame 2D quadratic interpolation on free-flight motion, and the gain concentrates in disoccluded regions.

The scope is deliberately limited. The method applies to rigid or near-rigid objects in free flight between contacts, such as sports, juggling, falling objects, splashing drops and bouncing balls. Aerodynamic drag, spin and self-propelled motion are deviations the system must detect rather than assume away.

Research Program

The system has four parts. The first is a ballistic segment detector. It runs point tracks and instance masks over the video, fits both a constant-velocity and a gravity-constrained 3D model to each object track, and keeps segments where the gravity model wins by a residual-based model-selection criterion. Bounces are detected as residual spikes and split the track into separate arcs.

The second part is the scale and focal solver. It fits $X_0$, $v$, the gravity direction and $\lambda$, optionally with focal length, by robust nonlinear least squares. All ballistic segments share one scale, one focal length and one gravity direction. Pooling many arcs is what makes the estimate stable. The accuracy should behave predictably. Acceleration is estimated from second differences of positions, so relative scale error should scale roughly as $\sigma/(a_{px} T^2)$. Here $\sigma$ is tracking noise in pixels, $a_{px}$ is the object's apparent acceleration in pixels per second squared, and $T$ is arc duration. We will verify this dependence on synthetic data, because it determines when the ruler can be trusted.

The third part is metric scene lifting. We align a temporally consistent relative depth estimate to the object's recovered metric depth at the frames where the object sits in front of known scene points. This parallels the least-squares alignment used with sparse LiDAR in Zablocki's LiDARTouch and with visual-inertial points in Wofk et al.. Our anchor, though, is physical dynamics rather than a sensor. Crowd4D is the closest analogue in spirit: Crowd4D resolves scale by forcing human feet onto reconstructed terrain. It relies on a human body template and contact constraints, whereas we rely on a law of motion that holds for any object in free flight.

The fourth part is the interpolation renderer. The background is a single metric point layer accumulated across all frames of the clip, which is possible because every frame now shares one metric scale. The flying object is a textured point set moved along its fitted parabola. For intermediate time $t$, we render both layers and composite them. Pixels uncovered along the true curved path are filled from background observations in other frames. A small learned refinement network then cleans up blending seams. It sees the disocclusion mask explicitly, following the warp-then-inpaint split of Balakrishnan et al.. No large generative model is involved; background that is never observed in the clip is left to the refiner and reported separately.

This design deliberately differs from the connected work in the file. Surface-Aware Feed-Forward Quadratic Gaussian also lifts frames to 3D for interpolation. Its authors state that it assumes linear motion between the 3D surfaces, and it works in unscaled space. We keep its 3D idea but supply the curvature and the scale. FreeOrbit4D smooths object motion with a Kalman filter in relative-scale space and fills disocclusions with multi-view diffusion. We use an explicit physical trajectory and reuse observed background. The ordinal depth loss in MoDGS is useful here as a negative example. It gives up metric scale to gain training stability, which is exactly what makes gravity unusable as a constraint in that pipeline.

Baselines

The naive reference point is linear 2D interpolation: bidirectional flow warping with constant velocity. It shows how much error the linear motion assumption causes on free-flight clips. Every method sees the same frames, subsampled from high-frame-rate footage.

The strongest four-frame competitor is Quadratic Video Interpolation, which estimates per-pixel acceleration from four frames. It reports PSNR 31.27 on GOPRO against roughly 29.5 for linear competitors, and an average feature-point shift of 0.97 against SepConv's 1.79. For the reproduction, we give it four frames while our method receives two. Beating it with half the input on free-flight clips, measured by lower object-center trajectory error and higher PSNR inside disocclusion masks, is the headline test.

The strongest two-frame non-linear competitor is BiM-VFI. The file contains no reported number for it; no reported result found on Lacuna for BiM-VFI on SNU-FILM or GOPRO. We will run its public model on identical inputs.

For large motion, Surface-Aware Feed-Forward Quadratic Gaussian reports PSNR 31.33 on X-Test-L 2K against SGM-VFI's 30.39. We will use it on our ballistic suite as the strongest 3D-lifting baseline.

All learned baselines use released weights. Each is also fine-tuned on our synthetic ballistic training split with the same number of GPU hours our refiner receives, so that data exposure does not explain any gap.

For scale recovery, the strongest baseline is Metric3D v2, which makes zero-shot metric depth predictions after normalizing images by focal length. We evaluate it twice, once with true focal length supplied and once with focal length estimated or missing. No reported result found on Lacuna for Metric3D v2 on our evaluation data. Our method beats it if its scale error is lower when focal length is unknown, and if it matches or complements Metric3D v2 when focal length is known.

Two ablations isolate the key ingredient. The first keeps our 3D pipeline but fits acceleration freely, as an unconstrained quadratic from four frames, instead of fixing $\|g\|$. This separates the contribution of known gravity from that of 3D lifting. The second runs our renderer at the correct 3D shape but in relative scale, which measures what metric grounding itself buys.

Why not just…?

Why not just use Metric3D v2 for scale? This is a real threat when the camera's focal length is known and the scene resembles the model's training data. Metric3D v2 makes its metric output conditional on focal-length metadata, and without it that output cannot be resolved zero-shot. Its per-image predictions also carry no guarantee of temporal consistency in scale. Gravity supplies scale from dynamics alone. The scale-error comparison with and without metadata settles the question. If Metric3D v2 with metadata wins outright, the proposal narrows to metadata-free video and to using gravity as a consistency check, and we will say so.

Why not just fit a 2D parabola in image space with four frames? For arcs roughly parallel to the image plane, a 2D quadratic is almost exactly what QVI already does, and it may match us. When the object moves toward or away from the camera, perspective division turns the 3D parabola into a non-polynomial image curve. A 2D quadratic also cannot use two-frame input, because the pixel-space acceleration depends on unknown depth. We will stratify results by the fraction of motion along the depth axis and report where the 2D model is adequate. We expect our advantage to vanish in the fronto-parallel bin and grow with depth motion.

Why not just fine-tune a learned two-frame interpolator on ballistic clips? A network could memorize typical arcs. But BiM-VFI's own analysis shows that two frames leave the intermediate position ambiguous, so the network has to guess an average over plausible arcs. Known $g$ at metric scale removes that ambiguity rather than averaging over it. The fine-tuned BiM-VFI baseline is exactly this alternative, with a matched compute budget. If it closes the gap on held-out arc speeds and camera distances, the physical prior is unnecessary.

Proposed Experiments

We will build a synthetic suite of physically simulated throws, drops and bounces. It will vary object size, drag, depth motion, camera focal length and frame rate, and include ground-truth metric depth and intermediate frames. This suite supports the error-scaling check and the training of the refiner.

For real data, we will extract free-flight clips from the 240 fps GOPRO and Adobe240 footage used by Quadratic Video Interpolation. We will subsample these to 30 fps and treat the held-out frames as interpolation ground truth. We will report PSNR over the full frame and inside disocclusion masks, together with object-center trajectory error.

Because these datasets lack metric ground truth, we will add a small phone-captured set of balls of regulation size thrown in scenes with surveyed distances. The known ball diameter gives an independent check on the recovered scale.

The decisive checks come in two parts. The first tests the scale solver. Its relative scale error should follow the predicted $\sigma/(a_{px}T^2)$ trend on synthetic data, stay within a few percent on real regulation-ball clips, and beat metadata-free Metric3D v2. The second tests the interpolator. With two frames, it should reduce trajectory error and raise disocclusion-region PSNR relative to four-frame QVI on clips with substantial depth motion. The known-gravity ablation should account for most of that gain.

Two failure analyses round out the study. Light objects such as balloons and shuttlecocks show where drag breaks the ruler, and spinning objects test the rigid-texture assumption. The detector should flag both cases rather than silently mis-scale them.