Inferring Mass and Friction Through Hidden Collisions

When two objects collide, the collision reveals their hidden physical properties: how heavy each one is relative to the other, and how bouncy the contact is. Current methods that recover such properties from video assume every frame is visible. Current trackers that follow objects behind occluders assume the objects just keep coasting. So when a collision happens behind an occluder, both fail. The estimator never sees the contact, and the tracker projects both objects along straight lines that are now wrong.

Our central observation is that the missing frames are less of a loss than they appear. Conservation laws constrain totals, not moments. Total momentum before and after a hidden impact must match, and the energy lost must be consistent with a restitution coefficient. Sliding friction leaves a deceleration signature in the visible stretches on either side of the gap. So the velocities of objects entering and leaving an occluded region carry enough information to bound their mass ratio and restitution even though the contact itself was never observed. We propose a factor graph that encodes this. A factor graph is a probabilistic model solved by least squares, in which unknown states are linked by soft constraints. Ours treats the occluded frames as free unknown states tied together by physics constraints. It treats each object's mass, friction and restitution as unknowns shared across the whole sequence. It handles the one genuinely discrete question, whether and which contact happened in the dark, by solving a small set of competing graphs. The output is a multi-modal belief over what happened behind the occluder, over each object's physical parameters, and over which re-emerging object is which.

Core Claim

Latent physical parameters stay identifiable across occlusion gaps that contain unobserved contacts, as long as the bodies involved are seen before and after the gap. A model that reasons over structured contact hypotheses and applies conservation constraints across the gap will recover these parameters. Two alternatives will not. The first fills in the missing frames and then runs a standard estimator. The second trains a neural estimator on masked inputs. The same reasoning resolves an identity question that appearance-based trackers cannot answer. When two identical-looking objects enter an occluder and two emerge, the assignment of identities that respects momentum conservation under the inferred masses is usually the only physically consistent one.

This is a different contribution from the closest existing work. ExPhy introduces PhyODE, a physical-property estimator coupled to a differentiable physics engine and a residual neural ODE. It shows that explicit continuous estimates of mass, friction and restitution make long-horizon forecasts more robust out of distribution. However, it requires clean 3D coordinates over the entire observation window and produces a single deterministic rollout. ComPhy infers discrete mass and charge labels from fully visible interaction videos. Both papers motivate the proposal: they show that property estimation helps prediction, but only under full observability.

On the tracking side, three papers handle occlusion but model no physical parameters. Navigation under uncertainty keeps Gaussian belief slots for hidden agents under switching linear dynamics. Scene Informer infers hidden agents from the reactions of visible ones. Occlusion Geodesics routes tracks through occluded space using a reachability cone set by pre-occlusion velocity. All three treat objects as purely kinematic, so none can tell whether a hidden impact changed an object's course.

The identifiability result of Mingyu Ding constrains what we can promise. An object that never interacts has unidentifiable latent parameters. Our method must therefore report wide posteriors in exactly those cases, rather than confident guesses.

Research Program

The model has three kinds of unknowns. The first is per-frame object states $x_{i,t}$ (position and velocity of object $i$ at time $t$), including frames where the object is hidden. The second is per-object parameters $\theta_i = (m_i, \mu_i, e_i)$ for mass, friction and restitution. The third is a discrete contact hypothesis $h$, which records which pairs of objects (or object and wall) touched during each gap, and in what order. For a fixed $h$, we solve

$$ (X^*, \theta^*) = \arg\min_{X,\theta} \sum_{t \in \text{vis}} \|z_{i,t} - x_{i,t}\|^2_{\Sigma_z} + \sum_{t} \|x_{i,t+1} - f_h(x_{\cdot,t}, \theta)\|^2_{\Sigma_f} + \sum_{\text{gaps}} \|r_{\text{cons}}(X, \theta; h)\|^2_{\Sigma_c} $$

Here $z_{i,t}$ are observed positions, available only in visible frames. The function $f_h$ is one step of rigid-body dynamics with sliding friction, including an impact at a free contact time when $h$ says one occurs. The term $r_{\text{cons}}$ collects momentum and restitution residuals that link the states just before and just after each hypothesized impact. The $\Sigma$ terms are noise covariances. For two bodies colliding in a hidden frame, momentum conservation reads

$$ m_1(v_1^+ - v_1^-) = -m_2(v_2^+ - v_2^-), $$

where $v^-$ and $v^+$ are velocities just before and after impact. Friction-corrected extrapolation from both sides of the gap estimates these velocities. This equation pins down the mass ratio $m_1/m_2$ and is the source of identifiability through occlusion.

Treating hidden states as free variables with soft dynamics residuals follows the tightly coupled smoothing approach of LIO-SAM. There, inertial constraints carry the state through stretches where LiDAR gives nothing. We reuse the idea of solving over a sliding window with incremental smoothing, but apply it to external objects rather than the ego vehicle. Relative to fitting a simulator by rolling it forward from the start, known in optimal control as single shooting, this "multiple shooting" form avoids a problem specific to contact. Small parameter changes flip whether a collision happens at all, which makes a single-shooting loss landscape discontinuous. In our form, each segment between contacts is optimized locally.

Multi-modality enters through $h$, not through the least-squares solver. This sidesteps the known weakness of factor graphs, whose Gaussian residuals cannot represent branching outcomes. We enumerate hypotheses only for objects whose kinematic reachable sets overlap during a gap. This keeps the count small: at most a handful per gap in scenes of 3–8 objects. Each hypothesis graph is solved and scored by a Laplace approximation to its marginal likelihood, which is a Gaussian fit around each solution. Normalizing these scores gives a mixture over hidden events, parameters, re-emergence identities and futures. Forecasts roll each weighted mode forward, so we report a distribution in the sense of Trajectron++. The difference is that our modes come from kinetic parameters and hidden events, not from learned intent codes.

The perception front-end is deliberately simple, because the question is about inference, not detection. In the primary setting, the inputs are ExPhy's ground-truth 3D coordinates with occluded frames deleted. In the secondary setting, the inputs are object masks from ComPhy videos with rendered occluders, passed through the same detector for all methods.

Baselines

The naive reference point is constant-velocity bridging of each gap, followed by the ExPhy property estimator on the bridged track, with nearest-position identity matching at re-emergence. This is what a standard tracker feeding a standard estimator does today. The occlusion papers above argue that such extrapolation drifts (Navigation under uncertainty), and we expect it to fail specifically on gaps that hide contacts.

The strongest existing method is PhyODE from ExPhy, run in two forms. The first is impute-then-estimate: missing frames are filled by linear interpolation and, separately, by a learned reconstruction in the spirit of the wavelet network of Physics-Informed Trajectory Prediction under Missing Observation. The second is PhyODE retrained on masked inputs with an explicit visibility flag. Both forms get the same training scenes, the same occlusion generator, and a hyperparameter search budget equal to the one we spend tuning our noise covariances.

As a reference point, ExPhy reports that PhyODE reduces ADE by 33.1% and FDE by 31.0% relative to its strongest baseline on the ExPhy-C out-of-distribution split. The file gives no absolute values, and we found no reported absolute property-estimation error on Lacuna for PhyODE. We will therefore re-run it ourselves under full visibility to anchor all comparisons. For discrete properties on ComPhy, the reference is CPL's reported "over 90%" mass and charge identification accuracy under full visibility (ComPhy).

A system-identification baseline fits a differentiable rigid-body simulator to the visible frames by single-shooting gradient descent from many random restarts. It receives the same wall-clock budget as our full hypothesis enumeration. This isolates whether the factor-graph formulation matters, or whether "fit a simulator" is enough.

Two ablations isolate the key ingredients. The first removes the conservation residuals and replaces discrete hypotheses with a single graph using a smooth soft-contact penalty. This tests whether explicit hidden-event reasoning is needed. The second keeps hypotheses but drops the shared parameters $\theta$, so dynamics use population-average physics. This tests whether per-object kinetics, rather than kinematics alone, drives the identity and forecast gains.

To count as beating PhyODE, the method must meet two conditions on the stratum of gaps that contain a hidden contact. Mass-ratio and restitution error must be lower than both PhyODE variants, with non-overlapping confidence intervals across seeds. Performance on fully visible scenes must not degrade.

Why not just…?

Why not just impute the missing frames and run PhyODE? This is a real threat when the gap contains no contact. Interpolation is then nearly exact, and our method should add little; we expect parity there and will report it. When a collision is hidden, interpolation draws two straight lines where there should be a kink. The estimator then sees either no interaction or a spurious one, which is exactly the shortcut regime ExPhy warns about. The stratified evaluation below settles this directly. If imputation matches us on the hidden-contact stratum, the proposal's claim is wrong.

Why not just train a neural estimator on masked inputs and let it learn to fill the gaps? This is the most serious alternative in-distribution. A large enough network may learn the momentum shortcut implicitly. ExPhy shows that black-box forecasters exploit kinematic shortcuts that collapse when physical parameters shift out of distribution. We will therefore test on ExPhy-B, which uses out-of-range parameters, with occlusions applied. We will also measure calibration, since a masked network has no principled way to widen its uncertainty in the non-identifiable cases of Ding. If the masked network matches us on ExPhy-B and is calibrated, the structured approach loses its justification.

Why not just use a particle filter or plain simulator fitting instead of a factor graph? Particles handle multi-modality natively, but they must sample over continuous masses, frictions and restitutions for every object, as well as contact timings. Particle depletion is likely exactly where the posterior is sharp, namely after an informative hidden impact. Simulator fitting by single shooting faces the discontinuity described above. Both are included as compute-matched baselines, so the comparison is empirical rather than asserted.

Proposed Experiments

We build ExPhy-Occ from the ExPhy PyBullet scenes. Static and moving occluders are placed so that three factors vary independently: gap duration, the fraction of each track hidden, and whether the gap contains zero, one or several contacts. Occlusion is applied to held-out scenes from ExPhy-A (in distribution), ExPhy-B (parameters out of distribution) and ExPhy-C (initial states out of distribution). We also render occluders into ComPhy test videos for a perception-in-the-loop check with discrete property labels.

The metrics are relative mass-ratio error, absolute friction and restitution error, and hidden-contact detection accuracy (whether the top hypothesis matches the true event). Tracking is measured by identity accuracy at re-emergence, reported separately for pairs of visually identical objects. Forecasting is measured by minimum-over-K ADE/FDE and negative log-likelihood. Calibration is measured by coverage of parameter credible intervals.

The first decisive check is the hidden-contact stratum. Our gains over imputed and masked PhyODE should concentrate there and vanish on contact-free gaps. A uniform gain would suggest we are winning for reasons unrelated to the claim.

The second check is identifiability honesty. On gaps where an object never interacts, the posterior width over its mass should stay near its prior. Width should shrink sharply only when the conservation constraint is active. A method that is equally confident in both cases is guessing.

The third check tests identity at re-emergence. Two visually identical balls of different mass collide behind an occluder. Appearance and nearest-position matching are at chance by construction. Our identity accuracy should rise with the mass ratio, since unequal masses make the conservation-consistent assignment unambiguous.

The fourth check is robustness out of distribution. The advantage over the masked end-to-end network should widen on ExPhy-B.

The fifth check is the ComPhy transfer. With rendered occluders, heavy/light accuracy should stay closer to CPL's full-visibility level than the baselines do under the same occlusion.

Failing the first or second check would falsify the core claim. Failing only the fifth would limit the claim to settings where the objects themselves are tracked reliably.