A single moving phone camera cannot tell how far away a moving object is. A ball twice as far away and twice as large, moving twice as fast, produces the same pixels. Current methods that rebuild moving scenes in 3D from one video learn a free-form motion field, and that field is happy to slide the ball back and forth along each camera ray without paying any cost in image error. The ball does pay a cost in physics, though. A thrown object must fall at 9.81 m/s², and a bouncing object must touch the floor at the moment it bounces. Both facts are expressed in metric units and tied to static surfaces, and the static background, unlike the moving object, can be triangulated from camera motion. We propose to reconstruct moving objects by generating their motion from a learned physics model rather than from a free deformation field. The model is conditioned on a small latent code per object that absorbs unknown properties such as bounciness, friction, and softness. Gravity and contact with the reconstructed background then pin the object's depth and the scene's metric scale.
The claim, stated in two sentences, is this. In monocular video, the along-ray depth of a moving object cannot be identified from images alone, and it cannot be identified from smoothness either. It can be identified from known gravity plus contact with static geometry, provided the unknown material behavior is absorbed by a learned latent physical code rather than hand-set. We will test where this holds (thrown, dropped, bouncing, and sliding objects under low camera parallax) and where it does not (self-propelled agents such as people and animals, whose motion gravity does not govern).
Why the obvious combination is not enough
Three existing lines of work each hold part of this idea. Monocular dynamic Gaussian splatting represents a scene as a cloud of small colored 3D ellipsoids ("Gaussians") that are warped over time. Deformable 3D Gaussians learns that warp as a time-conditioned MLP. USplat4D observes that depth uncertainty along the optical axis is far larger than lateral uncertainty, and ties uncertain Gaussians to confident neighbors through rigid graph blending. Both methods are purely kinematic. They have no notion of acceleration caused by a force, so they cannot fix the depth of an object that is moving through otherwise empty space.
Learned physics simulators do reason about forces. 3DGSim learns particle dynamics rendered through Gaussians. FreeGave attaches a latent physics code to each Gaussian and decodes it into a divergence-free velocity. Both assume multi-view capture (FreeGave uses a 20-camera rig), so depth is already solved before physics enters.
The only work in the file that uses physics to resolve projective depth ambiguity is Lecomte et al.. They regularize 2D-to-3D liver registration with a hyperelastic finite-element model, which reduces error from 3.8 mm to 2.8 mm. That result motivates us: constitutive physics, not kinematics, removed unphysical along-ray distortion. But it relies on a hand-coded solver, a fixed X-ray geometry, and a pre-scanned template.
Our proposal differs from all three in the same way. Physics is not the output here (as in the simulators), and it is not a fixed prior (as in the finite-element work). It is the mechanism that makes monocular depth identifiable, and its unknown parameters are learned latents inferred from the video itself.
Core Claim and Mechanism
Consider an object trajectory $X(t)$ seen by a camera that barely translates. The scaled trajectory $sX(t)$, for any scale factor $s > 0$, projects to identical pixels. Scaling also preserves smoothness, so penalizing acceleration or jerk leaves this one-parameter family intact. Gravity breaks the family. Under free flight, $\ddot X(t) = g\,\hat{e}_g$, where $g = 9.81$ m/s² is the gravitational acceleration and $\hat{e}_g$ is the unit vector pointing down, and the scaled trajectory has acceleration $s\,g$. Only one $s$ is consistent with the known frame rate. Contact breaks the family a second way: at a bounce time $t_c$, the object's surface must lie on the background surface, whose depth is triangulated from camera motion up to the same global scale. The two cues are complementary. Gravity fixes metric scale even when the background is poorly reconstructed. Contact fixes depth even when motion is not ballistic, for example when an object slides on a table.
The difficulty is that real objects do not follow textbook trajectories. They bounce with unknown restitution, slide with unknown friction, and squash. A hand-written rigid-body simulator would force a misfit, and the optimizer would "explain" the misfit by moving the object in depth, which is exactly the failure we want to remove. This is where implicit physical representation earns its place. We pretrain an object-centric dynamics model in simulation, with the same perception/prediction split as Perception-Prediction Networks. That work showed that latent codes trained only to predict trajectories come to encode log-mass and restitution ($R^2$ of 0.90 and 0.80). The dynamics model maps an object's pose, velocity, latent code $z$, a local signed-distance query of the static background, and the gravity vector to the next state. At test time we invert it: we optimize each object's initial state, its code $z$, the global scale $s$, and the gravity direction to match the video through the renderer.
The objective is
$$ \mathcal{L} = \sum_t \big\| I_t - R\big(G_{\text{static}}, \{G_k, \; T_k(t; x_k^0, z_k, s, \hat e_g)\}_k, C_t\big) \big\|_1 + \lambda \sum_{k,t} \| \delta_k(t) \|^2 . $$
Here $I_t$ is frame $t$, $R$ is the Gaussian renderer, $C_t$ is the estimated camera, $G_{\text{static}}$ are the background Gaussians, and $G_k$ are the Gaussians of object $k$ expressed in that object's own frame. $T_k$ is the object pose produced by rolling out the learned dynamics from initial state $x_k^0$. The term $\delta_k(t)$ is a small residual deformation for non-rigid detail, and its penalty $\lambda$ keeps it from reintroducing the free along-ray slide.
Because each object's Gaussians live in its own frame, densifying Gaussians where a moving object has high image error adds geometry to that object rather than leaving the orphaned world-space clusters that RobustSplat documents. This partly defuses the tension between suppressing densification and the catastrophic pruning reported in Monocular Dynamic Gaussian Splatting: Fast, Brittle, and Scene Complexity Rules.
Research Program
The work has four stages. The first stage reconstructs the static background and cameras with delayed densification in the style of RobustSplat, keeping a single unknown global scale. It also segments moving objects using latent-code clustering in the manner of FreeGave's unsupervised motion segmentation, falling back to a 2D video segmenter if that fails.
The second stage pretrains the latent-code dynamics model on procedurally generated trajectories of rigid, elastoplastic, and cloth-like objects. These are rendered with randomized gravity direction, restitution, friction, and stiffness, similar to the scenes 3DGSim trains on. The model never sees the test objects.
The third stage runs the inversion, using a coarse-to-fine schedule. First we fit $s$, the gravity direction, and initial states with $z$ held at the prior mean. Then we release $z$. Finally we release the residual deformation. This staging replaces the brittle multi-stage curriculum of PHYCINE with a schedule driven by what each variable can identify.
The fourth stage is analysis. We characterize when depth is identifiable as a function of camera parallax, using DyCheck's effective multi-view factor from Monocular Dynamic View Synthesis: A Reality Check, and of the fraction of each trajectory spent in free flight or in contact. We also check whether the inferred codes track ground-truth restitution and friction.
Baselines
The naive reference point is Deformable 3D Gaussians: a canonical Gaussian set plus a time-conditioned deformation MLP with annealed temporal noise. We run it on identical frames, poses, and segmentation, at its published settings. It shows how much depth sliding an unconstrained warp produces.
The strongest existing method for monocular dynamic reconstruction is USplat4D. It is the strongest competitor on this specific question because it is the only method in the file that explicitly targets optical-axis ambiguity. The file reports that it reaches 19.63 PSNR on DyCheck at 2× resolution, against 19.32 for MoSca. We run its released pipeline with the same camera poses and foundation-model tracks we feed our method, and give both methods the same GPU-hour budget for hyperparameter search on a held-out validation split. Beating it means lower depth error on moving objects and higher held-out-view PSNR on physics-driven scenes. It also means not losing more than a fraction of a dB on DyCheck iPhone scenes dominated by self-propelled humans, where we expect no gain.
A depth-prior baseline initializes and supervises the dynamic Gaussians with per-frame Depth Anything V2 depth, scale-aligned to the background. This mirrors the monocular branch of DepthSplat and the casual-video setting of MoDGS. We give it the same segmentation and a temporal-consistency loss so that it is not handicapped.
An upper-reference baseline runs FreeGave with all cameras of the multi-view source datasets, which bounds what physics-aware reconstruction achieves once depth is observed. The file reports 35.019 PSNR for FreeGave on future prediction on its Dynamic Indoor Scene Dataset, against 29.745 for TiNeuVox. No reported result was found on Lacuna for any method on a single-camera version of that dataset.
Three ablations isolate the key ingredients. The first replaces the learned dynamics with a second-order smoothness penalty on free trajectories. The second removes gravity, the third removes contact, and together these separate the two identifying cues. A final ablation replaces the latent-code model with an explicit differentiable rigid-body simulator whose restitution and friction are optimized directly.
Why not just…?
Why not just penalize acceleration or jerk on the Gaussian trajectories? Smoothness is the cheapest physical prior and a reviewer's first suggestion. Under low camera parallax it cannot work, because scaling a smooth trajectory along the camera rays keeps it smooth and keeps the projection identical. The smoothness ablation tests this directly. On the SlidingCube and RotatingCube scenes from Monocular Dynamic Gaussian Splatting: Fast, Brittle, and Scene Complexity Rules, which vary camera baseline independently of object velocity, we predict that its depth error grows as the baseline shrinks while the full model's error stays flat. If smoothness matches us at wide baselines, that is expected. If it matches us at narrow baselines, our central claim is wrong.
Why not just use a monocular depth foundation model? Depth Anything V2 gives relative depth whose scale drifts from frame to frame, and that drift appears as an object inflating and shrinking along the line of sight. ObjectFusion shows that errors from lifting each pixel by its estimated depth smear along the optical axis. This baseline is a real threat on textured, near-field objects, and there it may win on per-frame depth. The comparison therefore reports depth error separately for small, fast, or poorly textured objects and for large textured ones. We also test the combination: foundation depth as initialization, physics as the constraint. If the combination is best, we will report that honestly as the recommended system.
Why not just use an explicit differentiable rigid-body simulator? For a rigid ball with a clean bounce, an explicit simulator with fitted restitution may do as well as our method, and we expect it to. The latent code should matter where the explicit model is misspecified: soft objects, spin-dependent bounces, cloth, and rolling with unmodeled friction. In those cases a misspecified simulator trades misfit for wrong depth. The explicit-simulator ablation, broken down by material class, decides this question. If the learned code adds nothing on deformable objects, we will narrow the claim to "gravity and contact identify depth" and drop the learned component.
Why not just use a 2D video generator? A generator does not give a 3D reconstruction, and 3DGSim reports that fine-tuned 2D video models produce levitating objects in counterfactual tests. We therefore do not treat video generators as a competitor for depth, only for future-frame prediction, and there we report them only if compute allows.
Proposed Experiments
The main benchmark converts multi-view physics datasets into honest monocular tests: the rigid, elastic, and cloth scenes used by 3DGSim, and the FreeGave-GoPro and Dynamic Indoor data. One smoothly moving camera serves as input. The remaining cameras serve only for evaluation, following DyCheck's protocol against camera "teleportation" and its co-visibility masks. Metrics are absolute relative depth error on moving-object pixels, error in the recovered metric scale, masked PSNR, SSIM, and LPIPS on held-out views, PCK-T for 3D correspondence, and future-frame PSNR beyond the observed window. We also report the $R^2$ between inferred codes and ground-truth restitution and friction, following the PPN analysis.
The controlled study uses SlidingCube and RotatingCube, re-rendered with gravity and a ground plane, and sweeps camera baseline. The real-world test has two parts. We run the DyCheck iPhone sequences, scored against LiDAR depth, as a no-regression check on human-dominated motion. We also capture a small new set of iPhone LiDAR videos of dropped, thrown, and bouncing household objects under nearly static handheld cameras.
Two checks decide the outcome. The first is the baseline sweep: if our depth error on moving objects stays roughly constant as parallax goes to zero while USplat4D, Deformable 3D Gaussians, and the smoothness ablation degrade, then physics is doing the identifying work. The second is the gravity and contact ablations on the real captures: the recovered metric scale should agree with LiDAR to within a few percent when free flight is observed, and should degrade gracefully, rather than collapse, when only contact remains.
References
- Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction
- Uncertainty Matters in Dynamic Gaussian Splatting for Monocular 4D Reconstruction
- Learning 3D-Gaussian Simulators from RGB Videos
- FreeGave: 3D Physics Learning from Dynamic Videos by Gaussian Velocity
- Beyond respiratory models: a physics-enhanced synthetic data generation method for 2D-3D deformable registration
- Unsupervised Learning of Latent Physical Properties Using Perception-Prediction Networks
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- Monocular Dynamic Gaussian Splatting: Fast, Brittle, and Scene Complexity Rules
- Intrinsic Physical Concepts Discovery with Object-Centric Predictive Models
- Monocular Dynamic View Synthesis: A Reality Check
- DepthSplat: Connecting Gaussian Splatting and Depth
- MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos with Depth Priors
- ObjectFusion: Multi-modal 3D Object Detection with Object-Centric Fusion