Collisions as Evidence Against Optical Flow in Hand-Object Video

Many video systems fit 3D hand and body meshes by trusting optical flow, the per-pixel motion field between consecutive frames, as if it were measured truth. Flow is least reliable in exactly the places where hands matter most: fingers wrapping around a handle, surfaces sliding past each other, and thin objects hidden behind a palm. This proposal uses a simple physical fact to catch those errors. Two solid surfaces cannot occupy the same space. If a flow vector, once lifted to 3D, would push a finger mesh into an object, that vector is suspect. We propose to treat the pull between "match the flow" and "do not interpenetrate" as a per-pixel signal of flow reliability. We then down-weight the offending pixels rather than letting the mesh fitter average the two demands into a compromise that is both off-track and physically implausible.

The opportunity matters because the two communities fix their halves separately. Flow-dependent pipelines have no way to discount corrupt flow. Collision methods clean up 3D meshes after the fact and never ask which image evidence caused the collision. Using physical violations to decide where the flow should be trusted connects the two. The standard occlusion check for flow cannot see one class of error: flow that is wrong but consistent in both temporal directions, for example object motion smeared onto adjacent finger pixels. A physics check can.

Core Claim

During mesh fitting, a flow pixel whose data term pushes the mesh against a non-penetration barrier is far more likely to carry a large flow error than a pixel whose term does not. This physics-conflict signal catches errors that forward-backward flow consistency misses. Reweighting flow by it improves hand pose, object pose and penetration depth together, whereas adding a penetration loss alone trades one against the other. A light network can also learn this signal from the fitting process and predict it in a single forward pass. That removes the per-frame optimization cost that makes current flow-and-pose loops slow.

Concretely, let $\theta$ be hand and object pose parameters. Let $u_p$ be the estimated flow at pixel $p$, and let $\pi_p(\theta)$ be the 2D motion of the mesh surface point visible at $p$, obtained by projecting the mesh motion into the image. Let $B(\theta)$ be a differentiable penetration barrier that is zero for penetration shallower than a margin $\delta$. The fitting energy is

$$E(\theta) = \sum_p w_p\, \rho\big(u_p - \pi_p(\theta)\big) + \lambda B(\theta),$$

where $\rho$ is a robust penalty, $w_p \in [0,1]$ is a per-pixel trust weight and $\lambda$ sets the barrier strength. The conflict score of pixel $p$ measures how directly its data gradient opposes the barrier gradient:

$$c_p = \max\!\Big(0,\; -\frac{\langle \nabla_\theta \rho_p,\ \nabla_\theta B\rangle}{\|\nabla_\theta \rho_p\|\,\|\nabla_\theta B\|}\Big).$$

Here $\rho_p$ is pixel $p$'s term, and $c_p$ is computed only when $B>0$. The weights are updated as $w_p \leftarrow \exp(-\beta c_p)$, with temperature $\beta$, and the fit alternates with these updates. The margin $\delta$, set to 1–2 mm, follows ContactOpt. ContactOpt argues that small interpenetration models skin compliance, and that zero-tolerance barriers cause floating. With the margin, real skin squish, which does appear in the flow, is not flagged as an error.

Relation to Prior Work

The closest work is Bootstrapping Human Optical Flow and Pose. It alternates between fine-tuning a flow network toward the projected motion of a 3D skeleton and refining the skeleton against the updated flow. It motivates this proposal by showing that flow and pose can correct each other. It differs in two ways: the body has no volume, so it cannot detect collisions, and it assumes the skeleton is the more trustworthy side, taking 20.5 seconds per frame to exploit that. We do not rewrite flow toward the mesh everywhere. We use physics to locate where flow is untrustworthy and leave the rest alone.

CLOAF integrates body motion as an ordinary differential equation whose trajectories cannot cross, which guarantees zero self-collision. It never looks at image pixels, and it cannot repair a pose that starts out interpenetrating. Its hard guarantee also sits in tension with the compliance argument above. We treat it as a strong post-processing competitor, not as our mechanism.

Physics-Aware Hand-Object Interaction Denoising supplies a technique we reuse. It defines a surface-normal penetration depth that stays informative for thin objects, where volume-overlap measures fail. That paper applies the depth as a post-hoc filter on tracked trajectories and never feeds violations back to the image evidence. We use its penetration depth as $B$. CPF and Grasping Field provide alternative differentiable contact and penetration terms of the same kind.

In-Hand 3D Object Reconstruction from a Monocular RGB Video combines contact and penetration losses with an implicit surface but uses no flow and assumes a rigid grasp. Our setting targets the opposite case: objects that rotate or slide in the hand, where flow is the natural tracking signal and also the most fragile one.

On the flow side, FlowCam, Video Decomposition Prior and Motion-Attentive Transition document the failure we target. Each consumes flow as hard evidence and degrades when it breaks under occlusion. None has a physical test for which flow to distrust.

Research Program

The first component is an evaluation asset. HO-3D provides ground-truth hand (MANO) and object (YCB) meshes and cameras for every frame. We render the ground-truth 2D motion of each visible surface point to obtain dense ground-truth flow on hand and object pixels. This lets us measure flow error directly rather than only its downstream effects. We run an off-the-shelf flow network of the RAFT family, a commonly used iterative architecture of the kind analysed in Disentangling Architecture and Training for Optical Flow. We then characterize where its errors fall relative to contact regions and occlusion boundaries.

The second component is the conflict-weighted fitter defined above. It initializes from a per-frame hand-object estimator and fits over short temporal windows. Its outputs are refined meshes and a per-pixel trust map.

The third component is amortization. The trust maps produced by the fitter on training sequences serve as pseudo-labels for a small head attached to the flow network's features. The head predicts flow reliability near hands in one pass. We test two questions. First, does the predicted map transfer to held-out sequences? Second, can a single feedforward fit weighted by the predicted map recover most of the gain of the iterative fitter?

A secondary check on 3DPW asks whether the signal generalizes from hand-object collisions to body self-collisions. There, flow on a forearm crossing the torso is a known failure.

Baselines

The naive reference fits meshes to unweighted flow with no barrier. It shows how much damage raw flow does in contact frames.

The strongest existing competitor for pose accuracy is ContactOpt applied as post-processing to the flow-fitted result. It is strong because it directly optimizes hand-object contact and, on HO-3D, reports hand error dropping from 57.7 mm to 48.1 mm MPJPE relative to its base estimator (ContactOpt). For physical plausibility, the strongest competitor is the neural physical-loss denoiser. It reports HO-3D MPJPE of 10.7 mm against 10.9 mm for TOCH and 11.4 mm for the raw tracker, and on GRAB it raises the plausible rate from 0.42 to 0.95 (Physics-Aware Hand-Object Interaction Denoising). These numbers come from their own input pipelines and are not directly comparable to ours. We will therefore rerun both methods on our flow-fitted outputs, using the same initializations, the same sequences and the same hyperparameter search budget as our method.

For the flow-and-pose loop, we reimplement Bootstrapping with a MANO hand in place of the skeleton. The file has no reported HO-3D or 3DPW numbers for Bootstrapping. Its reported figure is Human3.6M MPJPE improving from 54.07 mm to 53.15 mm.

On 3DPW we compare against CLOAF. CLOAF reports a 0.0% collision rate versus 39.2% for HMR 2.0 and 9.2% for optimization methods, and it improves P-MPJPE from 58.1 mm to 55.4 mm for EFT.

The key ablation keeps the identical energy, with the barrier on and every weight fixed at $w_p = 1$. This isolates reweighting from simply adding a penetration loss. A second ablation replaces conflict weights with forward-backward-consistency weights.

Beating the baselines means two things. First, lower hand MPJPE and object pose error than ContactOpt and the denoiser, at equal or lower penetration depth. Second, flow end-point error in contact regions that falls below both the raw flow and the Bootstrapping-refined flow.

Why not just…?

Why not just add a penetration loss to the fitter and let the optimizer balance it? This is the most natural alternative and the one the fixed-weight ablation tests directly. When the data term and the barrier disagree, a summed objective produces a compromise. Either the mesh follows bad flow partway into the object, or $\lambda$ is raised until the hand floats. ContactOpt describes this contact-versus-penetration tension as requiring delicate tuning. Reweighting removes the bad evidence instead of averaging against it. If the ablation matches our method across a swept $\lambda$, the reweighting idea fails and we will report that.

Why not just use forward-backward consistency to mask unreliable flow? This is a real threat, because consistency checks are cheap and catch true occlusions. Our claim is narrower: consistency cannot catch errors that agree in both directions, such as a finger pixel inheriting the motion of the object behind it. We will measure, against rendered ground-truth flow, what fraction of large errors each signal flags. The decisive quantity is the error mass caught by physics conflict but missed by consistency. If that residual is negligible, the proposal reduces to a consistency mask plus a collision loss.

Why not just run CLOAF or ContactOpt after fitting? Post-processing moves the mesh but leaves the corrupted flow in place. Anything else that consumes that flow, such as object tracking across slippage or video decomposition, still inherits the error. CLOAF also cannot recover from an initial pose that already interpenetrates, and it handles only self-collisions. The comparison in Baselines tests whether upstream reweighting beats downstream correction on pose. The flow-error metric tests the claim that post-processing does not repair the evidence at all.

Decisive Checks

The first check is diagnostic and runs before any downstream claim. On HO-3D contact frames, the conflict score $c_p$ should rank ground-truth flow error better than consistency does. Ranking quality will be measured as area under the precision-recall curve for pixels whose error exceeds a few pixels.

We will also report the main confound: conflict blaming correct flow because the mesh initialization is wrong. We will estimate this by injecting controlled pose noise into ground-truth initializations and measuring how often correct flow gets down-weighted. If blame mostly falls on correct flow under realistic initial errors, the signal is diagnosing the mesh rather than the flow, and we will say so.

The second check is downstream. Across HO-3D, measured by hand MPJPE, object pose error, penetration depth, solid intersection volume and contact-region end-point error, the conflict-weighted fitter should beat the fixed-weight ablation and both post-processors on at least one axis without losing on the others.

The third check is amortization. The single-pass predictor should retain most of the iterative gain at a small fraction of the per-frame time that Bootstrapping reports.

The 3DPW self-collision experiment tells us whether the idea is specific to hands or a general property of flow near contact.