Completing Hidden Geometry from How Objects Move

A depth camera sees only the front of an object. Shape completion networks fill in the back from category priors learned on CAD models, so a partially seen mug, box or toy gets a plausible but unchecked rear. Yet objects in the world rarely sit still. They slide, rock, topple and roll, and the way they do so depends on exactly the parts the camera cannot see. A box that tips toward the camera has its center of mass forward of its support edge. A bottle that rolls in an arc has a tapered hidden side. This proposal treats an object's observed motion as evidence about its unobserved geometry and mass. Among the many completions a visual model finds plausible, it keeps only those that, when simulated under ordinary rigid-body physics, reproduce the partial trajectory the sensor actually recorded.

The opportunity sits where three literatures miss each other. Completion networks such as PCN, RFNet and 3D MeanFlow process single static frames and know nothing of physics. Physical predictors such as Predicting the Physical Dynamics of Unseen 3D Objects and NeuMA assume complete geometry is given. The one work that uses physical evidence to refine completion, CLASP, requires the object to stay perfectly still while a robot touches it. Here motion is the signal rather than the nuisance, and the physical law (rigid-body dynamics with contact) is shared across categories. That is what should let the method generalize to object classes the shape prior never saw.

Core Claim

The trajectory of a moving rigid object depends on its shape only through a few quantities: mass, center of mass, the inertia tensor, and the geometry of whatever surface touches the ground or another body. We claim that fitting a simulated rollout to a partial observed trajectory pins down these quantities well enough to correct visual completions. The correction should be largest on the support side and in mass distribution, which are the parts that matter most for predicting what the object will do next. We also claim the gain persists on held-out categories, because the constraint comes from physics and not from the category prior.

We do not claim that motion recovers all hidden geometry. A decorative bump far from any contact barely changes the trajectory, and free flight reveals only inertia ratios. The proposal makes this limit measurable: we report gains broken down by event type and by surface region, not as one aggregate number.

Research Program

The input is a short single-view depth sequence of an object interacting with a support plane, plus the camera pose. The output is a distribution over completed shapes, densities and future motions.

The pipeline has three stages. First, a fast one-step generative completion model in the style of 3D MeanFlow proposes many candidate completions from the first frame. At roughly 5 ms per sample, hundreds of hypotheses are affordable. We also keep a latent-space version with a VAE prior, following CLASP's shape-sampling network, so that hypotheses can be refined by gradient.

Second, we register the visible points across frames to obtain the object's observed pose track wherever it is visible. Every frame then contributes free-space evidence: rays that pass through a region at some time rule out the object occupying that region at that time, once the region is transformed into the object's own frame. This generalizes CLASP's free-space loss from a static object to a moving one, and it already exploits motion purely kinematically.

Third, and this is the new ingredient, each hypothesis is scored by simulating it. Let $z$ be a shape latent, $\rho$ a low-dimensional density field (uniform, shell, or a few-parameter interior blob), $x_0$ the initial pose and velocity estimated from the first frames, and $S_t(z,\rho)$ the simulated object surface at time $t$. Let $V_t$ be the visible depth points at frame $t$ and $\Pi_t$ the operator that keeps only the simulated points visible to the camera. The fit objective is

$$ L(z,\rho)=\sum_t d\big(V_t,\ \Pi_t S_t(z,\rho)\big)+\lambda_{f}\,L_{\text{free}}(z)+\lambda_{p}\,\lVert z\rVert^2 , $$

where $d$ is a one-sided Chamfer distance from observed to simulated visible points, $L_{\text{free}}$ penalizes occupancy of carved free space across all frames, and the last term keeps $z$ near the prior. A sampling stage (cross-entropy search over the prior samples, run in Bullet) handles the discontinuities caused by contact. A short gradient stage through a differentiable rigid-body simulator then refines the survivors. The surviving weighted set is the posterior.

Contact-rich events are where identifiability comes from, so the data generator deliberately includes sliding under friction, toppling, rocking and rolling. Friction and restitution are fit jointly with geometry, because a wrong friction coefficient could otherwise be absorbed into a wrong shape. We extend to deformables only in a final stage, if at all. That would mean replacing Bullet with an MPM simulator and a material adaptor as in NeuMA, which also addresses the "hollow shell cannot be simulated" problem. The core study is rigid.

Baselines

The naive reference is complete-then-simulate. We complete the first frame with RFNet or 3D MeanFlow, assume uniform density, and roll it forward in Bullet with friction fit to the visible track. This is the pipeline a practitioner would build, and it shows what visual priors alone give. For context, RFNet reports a Chamfer Distance of 8.13 and a Fidelity Error of 3.35 on ShapeNet across 8 categories (RFNet). That setting is static and not directly comparable, so we retrain it on our partial-view split under the same data and tuning budget.

The strongest geometric competitor is temporal fusion plus free-space carving, our stage two alone. It aggregates every surface that becomes visible as the object moves and carves away space the object never occupied. It is the CLASP objective made dynamic, without simulation. CLASP's own result, reducing mean Chamfer Distance from 0.5 cm to 0.1 cm on its Shallow Cheez-It benchmark using contact and free-space constraints (CLASP), shows how much such constraints can give. Its hyperparameter search budget will equal ours.

The strongest predictive competitor skips geometry. A PointNet-plus-recurrent model in the style of Predicting the Physical Dynamics of Unseen 3D Objects is trained end-to-end on partial depth sequences to predict future poses. That paper reports position error under 1.2 cm and rotation error under 4 degrees over a 1-second rollout on leave-one-category-out splits, but with complete shapes as input. We retrain it on partial inputs with the same simulation data. We also report its complete-shape version as an oracle upper reference.

Two ablations isolate the key ingredient. In the first, we remove the simulated trajectory term while keeping fusion and carving. In the second, we feed the trajectory of a different object to the physics term. A real gain should vanish under this mismatch, and if it does not, the improvement is coming from regularization rather than physics. Beating the field means lower Chamfer on never-observed regions, lower center-of-mass and inertia error, and lower future rollout error than both fusion-plus-carving and the end-to-end predictor on held-out categories.

Why not just…?

Why not just complete the first frame and hand it to a physics engine? Visual completions are chosen for looking plausible, not for having the right support geometry or mass distribution. Rigid-body rollouts are very sensitive to center of mass near tipping thresholds, which is the core observation behind the Rempe et al. predictor. The complete-then-simulate baseline will show this directly. We expect its errors to concentrate in toppling and rocking events, and our breakdown by event type tests that expectation.

Why not just fuse all frames, since moving objects reveal more of themselves? This is the real threat. When an object rotates substantially, fusion may recover most of the shape, and physics then adds little. We do not argue around this. We put fusion-plus-carving inside our own pipeline and report the physics gain against the fraction of the surface that was never observed in any frame. If the gain is zero when nearly the whole surface was eventually seen, the method still matters only in the regime where hidden faces stay hidden. That regime covers sliding, partial rocking, and the always-occluded bottom and interior. If the gain is zero there too, the core claim is false.

Why not just predict future motion end-to-end and skip geometry? If the only goal is the next second of motion, an end-to-end model trained on enough data might match us in-distribution. Our bet concerns held-out categories and downstream uses. A geometry-and-mass posterior can be passed to a grasp planner or a different simulator, while a learned predictor's errors on unfamiliar shapes are unconstrained by physics. The leave-one-category-out comparison settles which bet holds, and we will report it even if the end-to-end model wins in-distribution.

Proposed Experiments

The main benchmark is generated, following the Rempe et al. recipe. ShapeNet objects are simulated in Bullet with randomized friction, restitution and non-uniform density, including hollow and weighted variants that are indistinguishable from outside. Single-view depth is rendered with sensor noise, and splits are leave-one-category-out. Real-world evaluation uses YCB objects, as in CLASP, filmed toppling and sliding with a depth camera. Ground truth comes from the objects' known meshes, and we measure mass and center of mass by balancing. Metrics are Chamfer distance split into ever-seen and never-seen regions, center-of-mass error, normalized inertia error, future position and rotation error over one second, and tipping-outcome accuracy.

The decisive checks are three. The first is whether the physics term reduces never-seen-region error and center-of-mass error on held-out categories beyond fusion-plus-carving, especially for toppling and rocking, and not for free sliding where we predict little gain. The second is whether the mismatched-trajectory ablation erases that gain. The third is whether, on interior-weighted objects, the method recovers density offsets that no surface-only method can detect. We will treat any of the following as a result worth reporting rather than hiding: a small gain on held-out categories, a gain that appears only for trajectories with contact events, or a confusion between friction and shape that the joint fit cannot resolve.