Fast 3D reconstruction models can now turn a short video into a moving 3D scene in under a second. They do it with a single pass through a trained network instead of hours of per-scene fitting. The scenes they produce are not physically consistent, though. Objects slide through tables, a gripper passes through the mug it is holding, and parts of the background drift even though nothing touched them. For robotics this matters, because these reconstructions are wanted for collision checking, plan verification, and learning from logged robot data. All three uses fail when the geometry ignores basic occupancy.
We propose a simple observation that changes the problem. In robot manipulation videos, one body in the scene is known exactly: the robot. Its mesh and joint structure are in its description file, and its joint encoders give its exact pose at every frame. We use this free, articulated 4D ground truth in three ways. It is an exclusion constraint, since nothing may occupy the robot's volume. It is a contact signal, since a gripper that stops closing before it is fully shut must be holding something. And it is a metric and causal anchor, since known link lengths fix scale, and objects that are neither touched nor supported by something moving should not move. We build these signals into the correction loop of a feed-forward reconstructor. They stay a correction signal inside the network, not only a training penalty and not an expensive optimization run after the fact.
Core Claim
Existing feed-forward 4D reconstructors, most directly DGS-LRM, predict deformable 3D Gaussians plus dense 3D scene flow from monocular video in about 0.6 s. Gaussians here are small colored blobs whose collective rendering reproduces the images. Scene flow is a per-point 3D motion estimate. These models are trained only on photometric and flow targets, so their outputs interpenetrate freely. Differentiable non-penetration methods do exist, but they share two limits. They depend on templates: Keep It SMPL uses capsule approximations of a human body model, CPF uses spring networks against known object CAD meshes, and Mueller et al. use Gaussian hand models. They are also solved by test-time optimization.
Our claim is that in robot-recorded video the robot supplies the template for free. Feeding a robot-derived physical residual back into a short recurrent refiner should reach most of the plausibility gains of test-time optimization at a small fraction of the latency. It should also reduce object trajectory error, not only cosmetic penetration. That last point is the non-obvious prediction. If it holds, the robot is helping with the ill-posed part of monocular dynamics, which is separating camera motion, scale, and object motion. It would not merely be cleaning up the output.
Research Program
The base model is a DGS-LRM-style transformer. It takes a monocular clip and, new here, the robot's joint trajectory. It outputs per-frame Gaussians, scene flow, and a soft assignment of each Gaussian to "robot", "static", or one of a few object slots.
The robot's Gaussians are not predicted. They are rendered in place from the robot description file through forward kinematics, the standard map from joint angles to link poses. UniGrasp similarly takes the robot description file as a direct network input. Anchoring the robot this way fixes metric scale and gives the camera-to-robot registration directly.
On top of this sits a refiner in the style of ReSplat. ReSplat runs a few recurrent update steps driven by 2D rendering-error maps, is 100x faster than per-scene 3DGS fitting, and gains 3.5 dB PSNR on DL3DV over single-pass models. We extend its error input with physical residuals. At refinement step $k$, each non-robot Gaussian $i$ at frame $t$ receives $d_{i,t}$, its signed distance to the robot mesh at that frame. Negative values mean the Gaussian is inside the robot. The update is
$$G^{(k+1)} = G^{(k)} + f_\theta\big(G^{(k)},\, e^{\text{photo}},\, \min(d,0),\, c,\, m\big),$$
where $G$ is the full set of Gaussian parameters and trajectories, and $e^{\text{photo}}$ is the rendering error. The contact indicator $c$ is on when gripper aperture stalls above its closed value. When $c$ is on, the network must place object mass inside the finger gap and make that object's motion follow the gripper rigidly. The motion-without-cause residual $m$ is the scene flow magnitude of Gaussians assigned to objects that are neither within a contact margin of the robot nor resting on a moving object.
Training runs in simulation, where ground-truth meshes exist. The loss combines rendering, flow, and a penetration term modeled on the contact-and-repulsion balance of CPF. Following ContactOpt, which showed that allowing 1–2 mm of penetration gives more stable gradients, we tolerate a small margin rather than enforcing strict zero penetration.
The contact indicator is the design's answer to the "equilibrium brittleness" problem noted across the contact literature, where over-penalizing penetration produces floating objects. Here attraction is not a tuned hyperparameter. The gripper's own sensing says when contact exists.
Reachability, the fourth direction, enters as the downstream use of the reconstruction. GaussianDream discards its Gaussian heads at inference, so it cannot check collisions online. We keep explicit geometry precisely so that candidate grasps and trajectories can be checked for collision and joint-limit feasibility against the reconstructed 4D scene.
Baselines
The naive reference point is DGS-LRM as published: a single feed-forward pass with no robot input and no physical terms. We retrain it on the same simulated clips with the same backbone size and training steps. This isolates the effect of our additions from any data advantage. It is the strongest published feed-forward 4D reconstructor in the file. It reports roughly 14.89 PSNR on DyCheck and 0.21 ATE-3D on PointOdyssey. There is no reported result on Lacuna for DGS-LRM on LIBERO, RoboCasa, or any robot-manipulation reconstruction benchmark, so all in-domain numbers will be produced by us.
The strongest competitor on plausibility is a hybrid of two published methods. DGS-LRM output is refined by test-time gradient descent on the same penetration, contact, and motion-cause energies we use. The energy follows CPF, which reports a 3.71 mm reduction in penetration depth and a 9.34 cm³ reduction in solid intersection volume on FHB. The soft margin follows ContactOpt, which reports MPJPE falling from 57.7 mm to 48.1 mm on HO-3D at about 62 ms per hand-object pair. We give this optimizer the same robot mesh and joint states and sweep its iteration count. That sweep produces a full quality-versus-latency curve, not a single point.
The per-scene optimization reference is Robot See Robot Do. It reports 7.5 mm tracking error at about 1.4 s per frame plus a 3-minute model build.
Beating these baselines means two things. First, our physical metrics must lie on or below the optimizer's quality-latency curve at under one second per clip. Second, our object trajectory error must be lower than naive DGS-LRM.
Three ablations isolate the key ingredient, each at matched compute. The first keeps the physical terms only as a training loss and removes them from the refiner input. The second keeps robot conditioning but removes the contact indicator. The third drops the robot input entirely while keeping the refiner.
Why not just…?
Why not just run test-time optimization on DGS-LRM's output? This is the real threat. For a system that already tolerates seconds of latency it may be good enough. The test is the quality-latency sweep in the Baselines section. If a few dozen optimizer steps match us at comparable wall-clock time, the refiner is not justified, and we would report that honestly. We expect it to fall short for two reasons. Optimization only moves Gaussians it is given, so it cannot fix mis-assigned object identity. And the energy's balance of attraction and repulsion is brittle without learned priors.
Why not just add the penetration loss at training time? A training-only penalty teaches the average case. At test time it gives the network no information about where the violations are in this particular clip. The first ablation measures exactly this. If it matches the full model, the recurrent feedback is unnecessary and the paper becomes a simpler result.
Why not just mask the robot out as a distractor and composite its mesh afterwards? Distractor filtering in the style of 3DGS-HPC would remove the robot. The robot would stop corrupting the reconstruction, but it would also stop constraining it. Compositing the mesh afterwards shows penetration without preventing it. The third ablation shows whether robot information helps object trajectories, which masking cannot.
Why not hand the reconstruction to a physics simulator? PhysGM offloads dynamics to a material point method simulator for passive single objects. Starting a simulator from already-penetrating geometry tends to launch objects rather than settle them. It also does not recover the observed motion. We compare against simulating from DGS-LRM's first frame on the ground-truth object trajectory error.
Proposed Experiments
Data. Training uses simulated manipulation from LIBERO and RoboCasa, rendered with camera motion, plus Kubric-style clips without a robot to preserve generality. The simulators provide object meshes, poses, and joint states. Evaluation is in-domain on held-out LIBERO and RoboCasa tasks and on RLBench as an unseen simulator. For real transfer we use DECO-50, which has dual-arm joint logs. It has no object ground truth, so there we measure robot-object penetration against the known robot mesh, gripper-contact consistency, and held-out-view PSNR.
Metrics. The primary metrics are penetration depth and solid intersection volume, defined as on FHB, for robot-object and object-support pairs. We also report a phantom-motion rate, the fraction of untouched objects with nonzero predicted motion. Supporting metrics are object ATE-3D, PSNR on held-out views, and latency per clip.
Decisive checks. The first check is whether object ATE-3D improves over naive DGS-LRM in frames where the robot is far from all objects. That would show the anchor resolves monocular scale and camera ambiguity, not just local clipping. The second is the latency-quality comparison against test-time optimization. The third is a downstream reachability check. We plan collision-free, joint-limit-respecting grasps against each reconstruction, execute them in the simulator, and count how many are actually feasible, comparing DGS-LRM, the optimizer, and ours. The fourth is a no-regression check on DyCheck with the robot input blank, confirming the model still behaves as a general reconstructor when no anchor is present.
References
- DGS-LRM: Real-Time Deformable 3D Gaussian Reconstruction From Monocular Videos
- Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image
- CPF: Learning a Contact Potential Field to Model the Hand-object Interaction
- Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera
- ReSplat: Learning Recurrent Gaussian Splats
- ContactOpt: Optimizing Contact to Improve Grasps
- GaussianDream: A Feed-Forward 3D Gaussian World Model for Robotic Manipulation
- Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction
- 3DGS-HPC: Distractor-free 3D Gaussian Splatting with Hybrid Patch-wise Classification
- PhysGM: Large Physical Gaussian Model for Feed-Forward 4D Synthesis