Sliding Along the Ray: Resolving Single-Image Depth with Supported Primitives

A single photograph cannot tell a small chair close to the camera from a large chair farther away. Every object in an image can be pushed along the camera's viewing rays and scaled in proportion without changing a single pixel. Monocular depth networks settle this ambiguity by learning pictorial habits such as apparent size and height in the frame. Those habits fail when the camera zooms or crops, and they produce scenes in which objects float above the floor or sink into it. This proposal exploits a constraint that pictures obey but networks ignore: indoor objects rest on something. If each object is simplified to a few volumetric primitives, here superquadrics (a family of smooth solids, ranging from boxes to cylinders to ellipsoids, set by a handful of parameters), two checks become cheap and closed-form. The first is whether an object touches its supporting surface. The second is whether its center of mass sits over that support.

The core move is to correct a depth estimate only along the directions the image cannot see. For each object we optimize a single scale factor about the camera center, along with a small slant correction. That leaves the object's image footprint exactly unchanged, while contact and static equilibrium with primitive-approximated supports determine where along the ray the object must sit. Geometric simplification is not used here as a representation for its own sake. It is the device that makes a physical prior computable from one view, and the prior is applied only in the ambiguous subspace, so it never conflicts with image evidence.

Core Claim

Take any object $k$ with 3D points $X_k$ reconstructed from a depth prediction. Replacing $X_k$ with $s_k X_k$ for any $s_k>0$ leaves its projection unchanged. This is the scale–distance degeneracy described by Learning 3-D Scene Structure from a Single Still Image, where along every pixel ray infinitely many 3D positions give the same image. We claim that a per-object correction restricted to this null space, chosen so that each object's primitive approximation is in contact with and statically stable on its support, will do three things. It will reduce object-level depth error, it will remove floating and interpenetrating geometry, and it will make depth predictions far more robust to zoom and crop than the pictorial cues of current networks. The objective is

$$ \min_{\{s_k,\delta_k\}} \sum_k \Big[ \lambda_c\, C\big(P_k(s_k,\delta_k), S_{\pi(k)}\big) + \lambda_e\, E\big(P_k(s_k,\delta_k), S_{\pi(k)}\big) + \lambda_p\, (\log s_k)^2 \Big]. $$

In this objective, $P_k$ is the superquadric set fitted to object $k$ after scaling by $s_k$ and applying a small slant rotation $\delta_k$. $S_{\pi(k)}$ is the primitive or plane that supports object $k$, and $\pi(k)$ assigns each object its support. $C$ is a signed contact gap that penalizes both floating and penetration. $E$ is the horizontal distance from the primitive volume's center of mass, under uniform density, to the convex hull of its contact region. The last term keeps $s_k$ near the network's original guess unless the physics term disagrees. Each object has only one or two degrees of freedom, so the problem is small and well conditioned. Supports form a tree rooted at the floor, which lets us solve them in order from the floor upward.

Relation to prior work

Three lines of work motivate the proposal. Primitive abstraction has worked before for monocular structure: Learning 3-D Scene Structure from a Single Still Image modeled scenes as planar superpixels in a Markov random field, and MonoPlane fits planes to depth and normals from foundation models. Both are purely planar, though, so they cannot represent volume, mass, or support. Volumetric primitives exist in Superquadrics Revisited, which parses isolated objects, and SUPERDEC, which decomposes scanned 3D scenes. Both operate after the 3D geometry is known and never use primitives to settle depth. Physical plausibility has been enforced by PhyRecon, which places a differentiable simulator inside multi-view neural reconstruction and raises the stability ratio on ScanNet++ from about 25–26% to 78.16%. PhyRecon depends on many calibrated views and reports unnatural bulges in occluded regions, where its physics loss is free to deform shape. We differ in two ways. We work from a single image, and we let physics move objects only along the unobservable ray direction, never reshape them. That restriction is designed to rule out the bulging failure.

Two works constrain the design. Exploring Geometric Consistency for Monocular 3D Object Detection shows that monocular detectors rely on apparent size and vertical position, and that they break when the camera intrinsics change under cropping or zooming. Our correction uses the intrinsics explicitly, so it should be immune to this failure. That gives a sharp test, described below. Towards Ambiguity-Free Spatial Foundation Model reports that larger depth models settle more rigidly on a single, sometimes wrong, hypothesis. This is evidence that scale alone will not remove the errors we target.

The direction of geometric simplification itself carries a warning. Engineering simulations that strip geometry lose the features that govern behavior, as when a 2D pipe idealization omits 3D auxiliary lines in Numerical Investigation on Thermally Induced Self-Excited Thermoacoustic Oscillations, or when interior mechanics are dropped in Pressure waves from air gun bubbles. The analogous risk for us is that a chair reduced to a single box gains a support footprint its legs do not actually have. We address this directly in the research program and test it in the experiments.

Research Program

The pipeline has four stages. First, a frozen monocular depth model predicts depth and an off-the-shelf instance segmenter splits the image into objects. Second, the visible points of each object are unprojected with the true intrinsics, and a small feed-forward network in the style of SUPERDEC predicts a parsimonious superquadric set. Because the network is trained on ScanNet meshes, it completes the unseen back of each object. Large planar regions such as the floor, walls, and tabletops are fitted as planes with GC-RANSAC in the manner of MonoPlane. Third, a support graph is inferred: for each object, the primitive or plane directly below its lowest visible contact rays. The objective above is then solved. Fourth, the dense depth of each object is rescaled by its $s_k$ and slant correction, with fine detail kept as a per-pixel residual relative to the primitive. Image fidelity therefore comes from the network, and only placement comes from physics.

The primitive simplification is adaptive. The parsimony weight controls how many primitives an object receives, and we lower it only at the bottom of an object, where contact is decided. Thin supports such as legs can then become separate primitives, while the body stays coarse. We will measure whether this adaptivity matters by comparing a single primitive per object against adaptive bottom refinement. Our hypothesis is that over-simplified footprints mainly inflate apparent stability rather than harm depth accuracy. If that holds, it is a useful negative finding about where simplification is safe.

Baselines

The naive reference point is the frozen monocular depth model with no correction. Here we use a recent large relative-depth model aligned to metric scale with the usual per-image median scaling, and a metric model, run on identical images and intrinsics. The file contains no NYUv2 numbers for Depth Anything v2, so we will measure it ourselves. The strongest published NYUv2 figure in the file is Relative Error 0.100 for Depth Map Decomposition for Monocular Depth Estimation, which factors depth into a normalized map, a mean, and a standard deviation. We include it as a trained specialist. IEBins, which refines adaptive depth bins iteratively, is the strongest bin-based competitor (its public code is ShuweiShao/IEBins). The file reports only its KITTI result, Abs Rel 0.050, so its NYUv2 number will be reproduced under the same evaluation crop. Our method is a post-hoc correction, so we apply it on top of every one of these backbones. Beating a backbone means lower object-level depth error and higher stability for the same backbone, with global Abs Rel no worse.

The most dangerous competitor is a heuristic: "snap to floor." Detect the floor plane, then rescale each object along its rays until its lowest point touches the plane, with no primitives, no stability term, and no support hierarchy. This is the ablation that isolates our ingredients. A second ablation fits SUPERDEC-style primitives to the unprojected predicted point cloud but applies no correction, which shows whether primitives alone add anything. A third lets the optimizer move objects in all six pose degrees of freedom plus scale, which shows whether restricting to the ray null space matters. Finally, a single-view adaptation of PhyRecon's physics loss, with the shape free to deform, tests whether shape bulging reappears. Every variant uses the same segmentations, depth backbone, intrinsics, and hyperparameter search budget, tuned on a held-out ScanNet split.

Why not just…?

Why not just snap objects to the floor plane? This is a real threat. On cluttered tables it may capture most of the gain. It cannot handle objects resting on other objects, such as a lamp on a desk on a floor. It also cannot catch an object that touches the floor at the wrong scale, for instance a chair whose visible front leg contacts the floor while its reconstructed back hangs in the air or sinks through it. Stability and the support tree are meant to catch exactly these cases. We will report results separately for objects resting on the floor and objects resting on other objects. If snap-to-floor matches us on both, the stability term is unnecessary and we will say so.

Why not just use a bigger or metric-trained foundation model? Towards Ambiguity-Free Spatial Foundation Model finds that larger models commit harder to single hypotheses. Exploring Geometric Consistency for Monocular 3D Object Detection shows that learned depth relies on cues that break under zoom. Our correction is applied on top of the largest available models, so the question is whether it still helps them. The zoom–crop experiment answers this directly.

Why not just fit primitives to the predicted depth, SUPERDEC-style? That approach inherits whatever placement error the depth network made: the primitives faithfully reproduce a floating chair. The fit-only ablation measures this. We expect it to improve shape completeness but not stability or object depth.

Why not just train with geometry-aware augmentation? The augmentations in Exploring Geometric Consistency for Monocular 3D Object Detection raised KITTI-to-nuScenes transfer from 10.23 to 19.40 mAP. That is strong, and they are complementary to our method. We will fine-tune a backbone with them and apply our correction on top. If the augmented backbone alone closes the zoom-robustness gap and reaches equal stability, our contribution narrows to stability alone.

Proposed Experiments

The main benchmark is ScanNet and ScanNet++ single frames, which provide ground-truth meshes. We measure per-object median depth error, per-object 3D IoU and Chamfer distance against ground-truth instances, and the stability ratio: the fraction of reconstructed objects that stay put when dropped in Isaac Gym. Stability is evaluated twice, once on the primitives and once on the full corrected meshes, so that stability manufactured by simplification is exposed. PhyRecon's multi-view 78.16% stability ratio on ScanNet++ is a reference ceiling, not a like-for-like comparison. NYUv2 provides the standard global Relative Error and RMSE, where we only require no regression from the 0.100 reported for Depth Map Decomposition. It also provides plane quality by Variation of Information, compared against MonoPlane's reported 0.907.

The decisive check is the zoom–crop test. We rescale focal length and crop off-center on NYUv2 and ScanNet frames, update the intrinsics correctly, and plot per-object depth error against zoom factor. The backbone should degrade because its pictorial cues shift. The corrected output should stay nearly flat, because contact geometry depends on intrinsics, not appearance. A flat curve for our method and a rising one for the backbones would show that the prior resolves the true projective ambiguity rather than fitting the dataset. If the curves are equal, the idea fails. The second decisive check isolates the null-space restriction. Measured as reprojection error against the original image and visually as bulging or tilting, the six-degree-of-freedom and shape-deforming variants should match the original image worse while gaining little stability over the ray-only correction.