A single camera image cannot tell a small car nearby from a large car far away. Every monocular 3D detector therefore carries a band of uncertainty along each object's line of sight. Physics offers a second source of evidence that current systems waste: two solid objects cannot occupy the same space, and over time they cannot pass through one another. This note proposes using that fact as evidence about depth, not as a cosmetic cleanup. The search is restricted to the one direction the image cannot see: sliding each object along its camera ray while rescaling it so its image footprint stays fixed. Inside that restricted space, a video of several interacting objects rules out most depth assignments. Any assignment that makes neighbors overlap in the ground plane is excluded, and so is any assignment that makes two objects swap front-to-back order while one is hiding the other.
The restriction to the image's blind direction matters because it removes a failure that others have documented. When networks are trained with collision penalties, they take the easy way out and shove objects apart until they no longer match the image (Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements). If every allowed move leaves the 2D projection unchanged, that escape route does not exist. Collisions can only be resolved by choosing among depths the image already considers plausible. Because each object then has only one unknown per frame, a depth bin along its ray, the whole multi-object, multi-frame problem becomes a small discrete inference problem. It runs in milliseconds, instead of the per-image optimization that takes tens of seconds in systems like PhySIC.
Core Claim
The claim is that non-interpenetration, applied across time, adds depth information in exactly the cases where monocular depth is worst: occluded and crowded objects. Enforcing it only within the image's depth-scale blind direction improves 3D localization without degrading 2D alignment. The key new ingredient is a pass-through constraint. Suppose two tracked objects' 2D footprints overlap continuously over a stretch of frames. Their depth order cannot flip during that stretch unless they separate laterally, because a flip would mean they passed through each other. This couples depth hypotheses across frames and across objects in a way that neither smoothness priors nor single-frame collision terms capture.
Where This Sits
Three lines of prior work each contain a piece of this idea.
The first line motivates the use of context. Relational monocular detectors show that neighbors help resolve depth. MonoPair predicts center-to-center offsets between adjacent cars and refines all depths jointly by least squares, improving KITTI hard-case AP from 8.65% to 10.42%. DDMP-3D passes messages between pixels conditioned on depth. BAAM attends over car-to-car and road-layout context. All three regress soft relational statistics within a single frame. None forbids overlapping volumes, and none uses time. We keep their premise that neighbors carry depth evidence, but we replace learned offsets with a hard geometric exclusion that holds regardless of training distribution.
The second line supplies the idea of physical constraints but also limits how we can use them. Interpenetration work shows that collision terms improve plausibility. Keep It SMPL uses analytic capsule overlaps. REMIPS uses generalized winding numbers on a 512-vertex proxy body for multi-person images. Single-image coherent reconstruction of objects and humans combines a signed-distance collision loss with depth ordering. PhySIC runs a joint contact and collision optimization against a static inpainted scene. Every one of these works on a single frame or a static background. The lazy-solution result from IHMR is the key constraint on our design: it rules out free 3D collision resolution and motivates our restriction to the ray. IHMR's own fix was factorized sequential refinement for two hands. Our restriction is a different, more principled way to keep collision handling from destroying image alignment, and it extends naturally to many objects and many frames.
The third line motivates the gap. Dynamic reconstruction systems handle motion but ignore occupancy. Track3R predicts point maps and trajectories from monocular video, and its points can freely pass through one another. MAGneT-3D shows that temporal monocular detectors overfit their training cameras. It fixes this with image-derived anchors, but it never reasons about whether its boxes overlap. CRISP is the closest in spirit, since it uses a moving human's contacts to repair static scene geometry from video. Its environment, however, is a static planar world, and the moving agent probes that world. It never constrains the depths of multiple moving agents relative to each other.
Formulation
Each tracked object $i$ at frame $t$ receives a discrete depth variable $k_i^t$ over the linearly increasing depth bins of CaDDN. The object's 3D box at bin $k$ is its detected box slid along the camera ray through its 2D center, with its metric size scaled proportionally to depth. Its image projection is therefore identical for every $k$. The detector's categorical depth distribution gives a unary cost. We then minimize
$$ E(k) = \sum_{i,t} -\log p_i^t(k_i^t) + \lambda_c \sum_{i<j,\,t} O(b_i^t, b_j^t) + \lambda_g \sum_{i,t} G(b_i^t) + \lambda_m \sum_{i,t} M(b_i^{t-1}, b_i^t) + \lambda_p \sum_{(i,j)\in\mathcal{E}} P_{ij}, $$
where the terms are defined as follows. $b_i^t$ is the box implied by bin $k_i^t$. $O$ is the ground-plane overlap area between two boxes. $G$ penalizes floating above or sinking below the estimated ground surface. $M$ penalizes world-frame accelerations that exceed a class-specific limit, after ego-motion is removed using the dataset's ego poses. $P_{ij}$ is the pass-through penalty on occlusion episodes $\mathcal{E}$: stretches of frames in which $i$ and $j$ overlap in 2D. It charges any depth-order flip within the episode that is not accompanied by lateral separation.
Each object contributes a chain of about 80 bins over time, with sparse links between objects. We solve this with loopy belief propagation, or with exact dynamic programming per chain inside block-coordinate descent. The output is a marginal over depth for every object and frame, so the method also produces calibrated depth uncertainty rather than a single collision-free point estimate. The same formulation carries over to articulated people. There, the ray variable is each person's root depth, with body scale tied to it, and $O$ is replaced by REMIPS-style winding-number overlap between 512-vertex proxy meshes.
Research Program
The work has three parts.
The first part builds the inference layer as a detector-agnostic post-hoc module. It consumes per-frame boxes, categorical depth distributions, and track identities from any monocular temporal detector.
The second part makes the module trainable. We backpropagate through the belief-propagation marginals, so the detector's depth head learns to produce distributions whose ambiguity the physics layer can resolve, rather than overconfident peaks it cannot correct. Training uses the depth-supervision losses the detector already has, applied to the refined marginals. Because the refinement can never alter 2D projections, this training cannot reproduce the lazy solution.
The third part is analysis. We measure how much each constraint shrinks the depth entropy of each object, stratified by occlusion level and crowd density. This tests the claim that the gains concentrate where monocular evidence is weakest.
Baselines
The naive reference is the base temporal detector's own output, taking the most likely depth bin with no refinement. We use MAGneT-3D because it is the strongest temporal monocular detector in the file and is designed not to overfit camera geometry. It reports nuScenes NDS rising from 28.5% to 32.9% in-domain, and zero-shot average NDS rising from 12.1% to 18.6% when trained on nuScenes and tested on Waymo, Lyft L5, and ONCE (MAGneT-3D). Our module wraps this same detector with identical weights, so any gain is attributable to refinement. Beating it means a statistically reliable NDS and translation-error improvement on the occluded subsets, with no drop on the unoccluded subsets.
The strongest relational competitor is a temporal re-implementation of MonoPair's pairwise least-squares graph on top of the same detector. It has learned neighbor offsets and uncertainty weights but no occupancy term. On single-frame KITTI val, MonoPair itself reports 10.42% $AP_{3D}$ on the hard subset at IoU 0.7. DDMP-3D reports 12.78% $AP_{3D}$ on moderate Car. These numbers are the single-frame references our frame-by-frame variant must match or exceed when built on a comparable base detector. We tune both relational variants with the same hyperparameter search budget on the nuScenes validation split.
Three ablations isolate the key ingredients. The first lets collision resolution move boxes freely in 3D instead of only along the ray, using the same weights and solver. It should reproduce IHMR's lazy-solution effect, showing up as increased 2D reprojection error and worse translation error. The second removes the pass-through term, leaving single-frame collision, ground contact, and smoothness. The third removes all collision terms, leaving ground contact and motion smoothness, which amounts to a strong physically motivated tracker-smoother. For people, the reference is REMIPS, which reports 138.3mm MPJPE on Panoptic Studio from single images. We compare against it run per frame, and against it followed by temporal smoothing.
Why not just…?
Why not just resolve overlaps afterwards with 3D non-maximum suppression or a physics engine? Non-maximum suppression deletes one of two overlapping boxes instead of relocating it, so it throws away true detections in crowds. Physics-engine projection pushes boxes apart in whatever direction relieves contact, which is exactly the unconstrained resolution that IHMR found destroys image alignment. CRISP also found that feeding raw monocular geometry to a simulator produced a 55.2% motion-tracking failure rate. Our free-3D ablation is effectively this alternative made differentiable. If it matched the ray-restricted version, our central design choice would be refuted.
Why not just smooth depth over time with a tracker? This is the real threat. On driving data, ground contact plus motion smoothness may already remove much of the ambiguity for unoccluded cars. We treat it as a first-class ablation, not a strawman. Our claim is narrower: when an object is heavily occluded for several frames, smoothing only interpolates, whereas the pass-through term uses the occluder's depth to bound the hidden object. If the collision-free ablation matches the full model on the nuScenes low-visibility subset, the pass-through idea fails. We will report that honestly and fall back to presenting the method as a cheap physical-consistency filter.
Why not just use a metric depth foundation model? PhySIC shows that DepthPro and MoGe alone left 641mm mean scene error and 0.09 contact F1 on PROX. Only joint collision and contact optimization brought these to 227mm and 0.51. MAGneT-3D also shows that learned depth cues collapse under new cameras. Foundation depth can serve as a better unary term inside our energy, and we will run that variant. The question we test is whether occupancy reasoning still adds information on top of the unary. We predict it does for occluded objects, whose visible pixels do not include their contact with the ground.
Why not just add a collision loss to the network? IHMR documents the lazy solution, and REMIPS needed proxy meshes just to make such losses affordable. Our trainable variant does add a collision-aware signal to training, but only through ray-restricted marginals. The free-3D ablation, trained end to end, directly tests the naive version.
Proposed Experiments
The main benchmark is nuScenes. We report NDS, mAP, and mean translation error, stratified by the dataset's visibility levels and by the number of annotated neighbors within 5m. Zero-shot transfer follows MAGneT-3D's protocol on Waymo, Lyft L5, and ONCE, testing whether a geometric prior transfers better than learned relational offsets. The single-frame variant, without motion or pass-through terms, runs on KITTI val. Because this setting has no temporal terms, it isolates what per-frame collision reasoning adds, and it can be compared against the MonoPair and DDMP-3D figures above.
We add three physical-consistency metrics. The first is the fraction of predicted box pairs that overlap in the ground plane; ground-truth boxes never overlap, which makes this a clean sanity check. The second counts pass-through events per track. The third is the change in 2D reprojection IoU relative to the raw detector, which is zero by construction for our method and nonzero for the free-3D ablation. For people, we run Panoptic Studio and CHI3D video sequences with proxy-mesh overlap, reporting MPJPE and interpenetration rates against per-frame REMIPS. For people as for vehicles, the comparison is stratified by occlusion. Runtime is reported per frame, with a target of well under the base detector's own inference time.
The decisive check is a single comparison: the full model against the collision-free tracker-smoother on the lowest nuScenes visibility level. A clear reduction in translation error there, together with no meaningful difference on fully visible objects, would confirm that interpenetration across time is depth evidence. A null result would falsify the core claim. A secondary check is that the free-3D ablation should show higher reprojection error and no better translation error than the ray-restricted version. That outcome would establish the null-space restriction, not the collision term by itself, as what makes physical constraints safe to use under monocular ambiguity.
References
- Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements
- PhySIC: Physically Plausible 3D Human-Scene Interaction and Contact from a Single Image
- MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships
- Depth-Conditioned Dynamic Message Propagation for Monocular 3D Object Detection
- BAAM: Monocular 3D pose and shape reconstruction with bi-contextual attention module and attention-guided modeling
- Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image
- REMIPS: Physically Consistent 3D Reconstruction of Multiple Interacting People under Weak Supervision
- SINGLE-IMAGE COHERENT RECONSTRUCTION OF OBJECTS AND HUMANS
- Track3R: Joint Point Map and Trajectory Prior for Spatiotemporal 3D Understanding
- MAGneT-3D: Monocular and Domain-Generalizable Temporal 3D Detection
- Contact-guided Real2Sim from Monocular Video with Planar Scene Primitives
- Categorical Depth Distribution Network for Monocular 3D Object Detection