Fixing Collisions Only Where the Image Is Blind: Ray-Constrained Depenetration for Single-View Scene Reconstruction

When a system reconstructs a room from one photograph, it has to guess how far away each object is and how big it is. The photograph cannot decide this. If you shrink a chair toward the camera and pull it closer by the same factor, it looks exactly the same in the image. Current single-image scene generators make that guess separately for each object, so chairs end up inside tables and sofas end up inside walls. The standard repair adds a collision penalty and pushes objects apart. That push moves them in arbitrary 3D directions, so it also moves them in the image, and the reconstruction no longer matches the photo it came from. Two goals that should not compete end up competing: agreeing with the photo and being physically possible.

This proposal separates the two goals by construction. Each object's placement is split into the part the image determines (where it appears and how it is oriented) and the single number the image cannot see: a joint depth-and-scale coordinate along the camera ray. Collision and contact terms may move only that number, and image terms own everything else. Because non-penetration now acts on one scalar per object, the occlusion order visible in the photo becomes a hard inequality ("the lamp is in front of the sofa"), floor support becomes an equality, and the whole depenetration problem becomes small enough to solve globally in well under a second instead of by fragile gradient descent. The claim is that this removes the floating-versus-fusion trade-off for most of the error that actually occurs, without giving up any image agreement.

Core Claim

Consider a pinhole camera at the origin. An object with center $c_i$, isotropic scale $s_i$, fixed rotation $R_i$ and canonical vertices $v$ has world points $x = c_i + s_i R_i v$. Replacing $(c_i, s_i)$ by $(\lambda_i c_i, \lambda_i s_i)$ for any $\lambda_i > 0$ maps every point $x$ to $\lambda_i x$. Every point therefore projects to the same pixel, so the rendered image, silhouette and occlusion-free appearance are unchanged. Writing $u_i = \log \lambda_i$, the image-invariant freedom of a scene with $n$ objects is exactly the vector $u \in \mathbb{R}^n$, apart from a global gauge fixed by the floor plane or a metric depth prior. We propose to solve

$$ \min_{u} \; \sum_{i<j} P_{ij}(u_i, u_j) \;+\; \alpha \sum_{(i,k) \in \mathcal{S}} D_{ik}(u_i,u_k)^2 \;+\; \beta \sum_i \big(u_i - \hat u_i\big)^2 \quad \text{s.t.}\;\; d_i(u_i) < d_j(u_j) \;\; \forall (i \prec j) \in \mathcal{O}. $$

Here $P_{ij}$ is the interpenetration volume between reconstructed meshes $i$ and $j$, computed from generalized winding numbers so that it works for meshes of any genus. $\mathcal{S}$ is the set of support relations (object on floor, object on object) with $D_{ik}$ the gap between supporting surfaces. $\hat u_i$ is a category size prior. $\mathcal{O}$ is the set of occlusion orderings read off the instance masks, with $d_i$ the depth of object $i$ in the overlap region. Each $P_{ij}$ depends on only two scalars, so pairwise penetration can be tabulated on a grid in $(u_i, u_j)$. The occlusion constraints form a partial order, which gives a feasible sweep from front to back. A small residual refinement then lets in-image translation move only against a reprojection penalty.

The central hypothesis is empirical and testable before anything is built: most of the placement error of an independent-object reconstructor lies along the camera ray. If it does, restricting the physics to that direction costs almost nothing in reachable accuracy and gains image fidelity and stability. If it does not, the idea fails early and cheaply.

Relation to Prior Work

The problem is motivated by SceneReGen. That system generates each object in its camera-relative orientation with a diffusion transformer (a generator that learns by gradually denoising random noise into samples) and then separately queries scene context for translation and isotropic scale. It reaches 3D bounding-box IoU 0.536 on 3D-FUTURE, and its authors state that objects are placed with no collision regularization and intersect furniture and walls. Its factorization is what makes our proposal possible: the quantities it estimates independently, translation and scale, are exactly the ones in which the depth-scale ambiguity lives. We use SceneReGen as the front end and change only how placement is corrected.

The closest methods that combine rendering-based analysis with interpenetration penalties handle the two signals differently. Copy-Transform-Paste optimizes the full rigid pose and scale of two given meshes using a weighted sum of CLIP scores from a differentiable renderer, a fractional soft-ICP contact term and a penetration term, with a phased schedule. Layout-your-3D applies a tolerant collision loss to Gaussian-splat envelopes placed with monocular depth. Both let collision gradients act in all directions and reconcile them with image terms through weights. We instead give the two signals disjoint degrees of freedom, act on exact mesh boundaries, and use occlusion order as a hard constraint rather than as a quantity the renderer must rediscover.

PAT3D settles text-generated scenes with a differentiable rigid-body simulator and reports 0% penetration ratio. It takes about 27 minutes per scene, and settling under gravity moves objects freely. It is our reference for what fully physical correction achieves and costs.

Three further lines of work constrain the design. ContactOpt and CPF refine hand-object poses with contact and repulsion terms after regression. ContactOpt discards the image during optimization, and CPF reports that dropping raw vision poses into a physics engine makes objects fly away. Both are evidence that post-hoc correction detached from the image is unstable or lossy. ContactOpt's finding that 1 to 2 mm of tolerated penetration improves hand fits tells us not to demand exact zero: we use a small tolerance band in $P_{ij}$. Keep It SMPL shows that analytic capsule proxies give smooth self-collision gradients, but also that proxies cannot represent concave contact. We therefore compute penetration on the real meshes, which the one-dimensional parameterization makes affordable. Pushing the Envelope for dense hand pose and the Analysis-by-Synthesis Transformer illustrate the blindness we exploit: silhouette and pixel losses give no signal along the line of sight.

Research Program

The first stage is a diagnostic. On 3D-FUTURE test scenes, we take SceneReGen's per-object placements and decompose each object's center and scale error, relative to ground truth, into the ray-scaling component and the orthogonal remainder. We report the fraction of error energy along the ray and the best 3D IoU reachable by an oracle that corrects only $u$. This oracle ceiling bounds everything that follows.

The second stage builds the solver. It has four parts: winding-number penetration volumes tabulated on a coarse $(u_i,u_j)$ grid and refined locally; occlusion constraints extracted from overlapping instance masks; support relations inferred from mask adjacency plus the floor estimate; and a global solve by front-to-back sweep followed by a few projected gradient steps. We will also ask whether the ranking of objects' depths is ever wrong in the initial prediction. If the ranking is wrong, a naive local optimizer must pass one mesh through another, and the sweep avoids that.

The third stage is a stress test outside furniture. On HO-3D we apply the same restriction to the relative hand-object depth coordinate, with a ContactOpt-style tolerance. The question is whether the idea transfers to articulated hands, where scale is roughly known and the invariance holds only approximately.

Baselines

The naive reference point is SceneReGen's output with no correction. On 3D-FUTURE its authors report bounding-box IoU 0.536, scene Chamfer distance 0.009 and F-score 89.50. No reported penetration or floating numbers for SceneReGen on 3D-FUTURE were found on Lacuna, so we will measure penetration volume, penetration ratio and support gap ourselves on its released outputs or a faithful rerun.

The strongest existing competitor is a joint weighted optimization in the style of Copy-Transform-Paste. It optimizes all of translation, rotation and scale per object, using a differentiable silhouette reprojection loss in place of CLIP (the image here is the target), together with fractional soft-ICP contact, the same winding-number penetration and the same size prior. It is run with its phased schedule on identical inputs. Its loss weights receive a tuning budget on a validation split equal to or larger than ours, and wall-clock time is matched. It is the strongest competitor because it uses every ingredient we use except the division of degrees of freedom. Copy-Transform-Paste reports 79.65% user preference for physical plausibility over geometric baselines on its own 50-pair set, but no reported result was found on Lacuna for it on 3D-FUTURE. Beating it means matching or exceeding its 3D IoU while achieving lower penetration and higher reprojection mask IoU at the same time. It must not be merely better on one axis at the cost of another; the comparison is a Pareto curve swept over its weights.

The second physical baseline is PAT3D-style simulation settling. Objects are placed from SceneReGen and settled under gravity in a differentiable rigid-body simulator, with semantic backpropagation replaced by a reprojection loss. PAT3D reports 0% penetration ratio and about 27 minutes per scene on its own 18-prompt benchmark. We compare on penetration, image fidelity and runtime.

The key ablation removes each ingredient in turn. First, keep our losses but let collision gradients act on all six pose dimensions plus scale; this isolates the ray restriction. Second, keep the ray restriction but replace the global sweep with plain gradient descent from the initial $u$; this isolates the ordering constraint and the global solve. Third, replace exact meshes with oriented bounding boxes; this isolates the value of exact boundaries.

Why not just…?

Why not just add a penetration loss to the existing joint optimization and tune the weights? This is the real threat, and the main comparison is designed to settle it. The file's evidence cuts both ways. Copy-Transform-Paste works reasonably well with this recipe, yet it still leaves interpenetration residuals and needs a phased schedule to avoid bad minima. The research file also names the floating-versus-fusion dilemma as an open problem for weighted repulsion. If the weighted baseline's Pareto front over its weights reaches our point, the restriction adds nothing and we will say so. Our prediction is that it cannot: any collision gradient component orthogonal to the ray must be paid for in reprojection error.

Why not just run a physics engine or a push-apart step afterwards? CPF reports that raw vision poses dropped into Bullet make objects fly away. The Copy-Transform-Paste user study found participants preferred joint semantic-geometric optimization over Shrinkwrap-style post-processing. PAT3D's settling costs about 27 minutes per scene. More basically, a free-direction push corrects the part of the scene the image already got right. The simulation-settling baseline measures how much image fidelity this costs.

Why not just train the reconstructor with a collision loss so no correction is needed? Feedforward learning has not closed the last-millimeter gap elsewhere. ContactOpt's optimization beat the learned RefineNet, and the feedforward scene generators evaluated in PAT3D exceeded 60% penetration ratio. Still, a collision-aware fine-tune of SceneReGen's translation-and-scale head is a legitimate alternative. We will run it at matched compute and also report it combined with our solver, since the two are complementary rather than exclusive.

Why not just use bounding-box or capsule proxies, which are cheap? Keep It SMPL shows that capsules work only for coarse limbs. SceneReGen's residual clipping involves thin, concave parts such as chair rungs and table legs, which boxes cannot represent. The one-dimensional parameterization is what makes exact mesh penetration affordable, and the box ablation measures what the proxy loses.

Proposed Experiments

The decisive first check is the error decomposition on 3D-FUTURE. If under roughly half of SceneReGen's placement error lies along camera rays, the oracle-$u$ IoU ceiling will sit close to 0.536 and the proposal should be narrowed or abandoned. If the error is dominated by the ray component, the oracle ceiling should rise well above 0.536. This follows from the depth-scale ambiguity being the one direction the image loss cannot supervise.

The main experiment runs every baseline and ablation on 3D-FUTURE scenes. We report bounding-box IoU, Chamfer distance and F-score against ground truth. We report penetration volume and penetration ratio, the latter following PAT3D's definition. We report mean support gap, to catch floating. We report mask IoU between re-rendered reconstructions and the input instance masks, to catch image drift. We report runtime per scene. Success means lower penetration than SceneReGen and the weighted joint baseline, reprojection mask IoU essentially equal to SceneReGen's (which our method preserves by construction), 3D IoU above 0.536, and runtime orders of magnitude below PAT3D's 27 minutes.

A robustness experiment perturbs initial depths to induce wrong front-to-back orderings. It measures how often gradient descent versus the global sweep recovers the correct order, which tests the claim about initialization sensitivity.

The transfer experiment on HO-3D refines hand-object relative depth. We compare against ContactOpt, which reports reducing MPJPE from 57.7 mm to 48.1 mm, on MPJPE, penetration depth and intersection volume. Here we expect a smaller gain, because hand scale pins the invariance only approximately. That result will show how far the idea reaches beyond rigid scenes.