Physics as a Depth Cue: Correcting Single-Image Scene Depth by Making Objects Touch Without Overlapping

A single photograph cannot say how far away things are. Every point along a camera's line of sight lands on the same pixel, so a depth predictor has to guess. Modern monocular depth networks guess well on average, but they still misplace whole objects relative to one another: a chair sinks into the floor, a lamp floats above a table, a sofa passes through a wall. We propose to treat these physical violations as evidence rather than artifacts to clean up later. The scene from one image is broken into a few hundred geometrically coherent patches. Each patch may slide only along its own camera rays, a move that leaves the image it projects to exactly unchanged. The slides are then chosen so that solid objects do not overlap and supported objects rest on their supports. Because this correction never changes what the camera sees, it targets only the part of the problem that the image cannot settle.

The core claim is that non-interpenetration and support contact, applied per patch and restricted to motion along rays, carry depth information that a strong single-image depth model lacks. The gain should be largest where depth models are weakest: occlusion boundaries, thin supports such as chair legs, and objects that are partly hidden. Restricting motion to rays is what separates this from ordinary post-hoc collision cleanup, which pushes objects apart in arbitrary 3D directions and so breaks agreement with the image. Working at the patch level, rather than per point or per voxel, is what keeps the pairwise physics affordable for whole rooms.

Where This Sits

The proposal generalizes one idea already shown to work for people. Body Size and Depth Disambiguation in Multi-Person Reconstruction from Single Images fits human body meshes from one image. It resolves the small-and-near versus large-and-far ambiguity by giving each person a free scale and requiring every person's feet to touch a single plane fitted to the floor. That paper motivates our approach: it shows that a physical contact constraint can fix depth. It also marks the limits we remove. It needs a flat floor, it models only human bodies, and it uses no penetration field between people and furniture.

PhySIC is the closest existing system. It builds a metric scene "scaffold", meaning a background point cloud obtained by erasing the person from the image, filling in the hole, and running the DepthPro and MoGe depth models. It then optimizes a human mesh against that scaffold using contact and an occlusion-aware penetration penalty. The difference matters: PhySIC treats the scene itself as fixed, rigid ground truth and moves only the human. We instead treat the scene's depth as the unknown and let physics among scene objects correct it. PhySIC's reported failures, such as warped geometry from the filled-in background and rigid-surface assumptions, are exactly the errors a scene-side correction could repair.

The penetration machinery is borrowed, not invented. Grasping Field penalizes points where two signed distance fields, functions giving each point's distance to a surface with negative values inside, are both negative. ContactOpt shows that allowing a small tolerated penetration of about 2 mm stabilizes optimization. We adopt both.

The patch representation comes from Superpoint Graphs and Superpoint Transformer (SPT), which group millions of points into homogeneous "superpoints" connected by an adjacency graph. Those works use superpoints only for labeling scans captured by depth sensors. We use them as the unit of depth correction on point clouds lifted from a single image.

Single-image and video reconstruction systems are the competitors. ISO predicts indoor voxel occupancy from one image. It weights how 2D features are spread into 3D using depth distributions derived from Depth-Anything. 3D Former, TransformerFusion and IOAR reconstruct scene surfaces from video. All four treat the scene as one monolithic surface or occupancy grid with no object-object physics, and the voxel-based ones are limited by grid resolution.

Core Mechanism

Start from one RGB image. A metric depth model gives a depth $d(u,v)$ at each pixel. This depth is lifted into a point cloud and split into superpoints $i = 1,\dots,S$ using SPT's partition. Image-feature boundaries are added to the partition, so that a cut appears wherever the image shows an object edge. Each superpoint gets one scalar unknown, a log-depth offset $\delta_i$. Every point of superpoint $i$ moves to $\mathbf{r}(u,v)\, d(u,v)\, e^{\delta_i}$, where $\mathbf{r}$ is the unit camera ray through pixel $(u,v)$. Pixel positions are therefore unchanged by construction.

A lightweight fusion network attends from each superpoint's geometric descriptor to image features. Geometric descriptors are planarity, normal direction, and size. Image features come from the region the superpoint covers, aligned by its known projection and so free of calibration drift. The network predicts three things:

  • a depth-uncertainty weight $w_i$;
  • for each adjacent superpoint pair, a contact probability $c_{ij}$ and a support direction relative to estimated gravity;
  • a coarse amodal thickness, meaning how far the object extends behind its visible surface.

The thickness turns each visible surface into a closed signed distance proxy $\phi_i$. This is needed because penetration cannot be measured between front-facing shells alone.

The offsets then minimize

$$ E(\delta) = \sum_i w_i \delta_i^2 + \lambda_p \sum_{(i,j)\in\mathcal{G}} \sum_{x} \big[\max(0,\, -\phi_i(x) - \tau)\cdot \max(0,\, -\phi_j(x) - \tau)\big] + \lambda_c \sum_{(i,j)\in\mathcal{G}} c_{ij}\, g_{ij}(\delta)^2, $$

where the terms are:

  • The first term keeps each superpoint near the depth model's prediction, more strongly where the predictor is confident.
  • The second term is a Grasping-Field-style co-occupancy penalty. It sums over sample points $x$ for superpoint pairs in the adjacency graph $\mathcal{G}$, with a ContactOpt-style tolerance $\tau$.
  • The third term pulls predicted support pairs together. Here $g_{ij}$ is the gap between the supported superpoint's lowest points and its support surface, measured along gravity.

The weights $\lambda_p$ and $\lambda_c$ balance the terms. A fully transparent, superpoint-based version of PhySIC's occlusion-aware rule zeroes the penetration gradient for points the camera never sees whose geometry is pure guesswork.

The graph is sparse, so the cost grows with the number of adjacent pairs rather than the square of the point count, and room-scale scenes remain tractable. The fusion network is trained on ScanNet with ground-truth depth as supervision, through the optimization unrolled for a fixed number of steps.

The floating-versus-clipping tension flagged in the literature is handled explicitly. Contact is enforced only where $c_{ij}$ predicts it, so a legitimately unsupported object is not dragged down onto whatever lies beneath it.

Research Program

The first question is diagnostic: how much of single-image depth error lives in per-superpoint offsets at all? On ScanNet frames, we fit the oracle offsets $\delta_i^\ast$ that best match ground-truth depth and measure how far the error drops. If oracle per-superpoint correction recovers little, the premise fails. We would then report that, and the work would shift to finer units.

The second question is whether physics can find those offsets without ground truth. We decompose the remaining gap by region type: occlusion boundaries, support contacts, thin structures, and open floor.

The third question concerns the amodal thickness predictor, which is the weakest link. We measure how sensitive the results are to thickness errors. We also compare learned thickness against crude fixed-thickness extrusion.

Baselines

The do-nothing reference is the raw metric depth model with no correction, lifted to points and meshed identically. We will run the strongest available predictor, DepthPro with MoGe detail as used in PhySIC, so that any gain is not simply a fix for a weak depth model. Beating it means lower depth error together with lower interpenetration volume and floating rate. A physics-only improvement that worsens depth does not count.

The strongest single-image competitor is ISO on Occ-ScanNet. It uses the same input regime and also combines Depth-Anything depth with image features. We will retrain it on the same training split with a matched GPU budget and compare in its own occupancy metrics as well as ours. No reported result was found on Lacuna for ISO's occupancy scores, because the file does not include them, so we will reproduce ISO rather than quote it.

As an upper reference from multiple views, 3D Former reports ScanNet mesh accuracy of 0.032 m, down from 0.055 m for prior work (Monocular Scene Reconstruction with 3D SDF Transformers). A single image is not expected to reach this. Closing a measurable share of the gap between the raw single-image depth and that number is the target.

A plane-snapping baseline adapts Body Size and Depth Disambiguation to scenes. It fits a RANSAC floor and gives each segment a free offset along its rays so that floor-touching segments meet the plane, with no penetration term.

Two ablations isolate the key ingredients:

  • Unconstrained 3D translation per superpoint under the same energy, which tests whether along-ray motion matters.
  • Per-pixel offsets instead of per-superpoint offsets, at matched compute, which tests whether the patch unit matters.

All methods share the same depth input, the same instance masks for evaluation, and the same hyperparameter search budget on a held-out validation split.

Why not just…?

Why not just use a better monocular depth model? This is the real threat. If the next depth model already places objects consistently, physics has nothing left to fix. The experiments settle this in three ways. We apply the method on top of the strongest available model. We report gains stratified by region type. We repeat the comparison across two depth models of different quality. If the gain shrinks to zero as the base model improves, the honest conclusion is that physics is a crutch for weak predictors. We would report that outcome.

Why not just add a penetration loss when training ISO or 3D Former? A training-time regularizer shapes average behavior. It cannot enforce constraints on a specific test image, and it acts on monolithic occupancy grids with no notion of separate objects. We include this as an additional baseline: ISO retrained with a co-occupancy loss on ground-truth instance labels. Our claim is that test-time, per-image correction along rays beats it on the penetration metrics and at least matches it on depth.

Why not just resolve collisions after reconstruction by pushing meshes apart? Free 3D separation can always remove overlap. It does so by moving objects to positions that no longer match the image, which trades physical errors for reprojection errors. The unconstrained-translation ablation measures exactly this. It should give similar penetration scores but worse depth and reprojection error.

Proposed Experiments

Training and evaluation use ScanNet with its reconstructed meshes and instance labels, and Occ-ScanNet for single-image occupancy (Monocular Occupancy Prediction for Scalable Indoor Scenes). Metrics fall into three groups:

  • Depth: absolute relative error.
  • Geometry: mesh accuracy and completeness against ScanNet surfaces, in 3D Former's units.
  • Physics: pairwise interpenetration volume between ground-truth instances, and the fraction of supported instances whose contact gap exceeds 2 cm.

The physics metrics are measured on the reconstruction, so they reward correct placement rather than mere repulsion.

The decisive check is the stratified comparison against the raw strongest depth model. Gains must appear at occlusion boundaries and support contacts, and must not be bought with depth regressions on open surfaces. A second decisive check is the oracle-offset study, which bounds how much any per-superpoint method can achieve. A third is a stress test on scenes with stairs and multi-level furniture. That is where planar snapping should fail and graph-based support should not.