Gravity as a Prior: Self-Weight Stiffness for Single-Image Reconstruction Under Noisy Masks

Most single-image 3D reconstruction pipelines work in two steps. A 2D model first cuts the object out of the picture (a segmentation mask), and a 3D stage then "lifts" that cutout into a shape. Both steps fail in predictable ways. The lifting step has to guess depth, because many 3D shapes produce the same picture. It also treats the mask as ground truth, so a mask that swallows a shadow creates floating "ghost" material, and a mask that misses a thin chair leg deletes the leg in 3D. We propose to borrow a tool from structural engineering to catch both failures. The reconstructed object sits on a visible floor and is pulled down by gravity. It must therefore carry its own weight to the ground. Floating blobs, severed legs and hairline connections all fail that test, and they fail it in a way that yields a smooth, usable gradient.

Concretely, we represent the object as a grid of continuous material densities, as in density-based topology optimization, the engineering method that decides where to put material in a part by relaxing "solid or empty" to a number between 0 and 1. We fit this grid to the image and to the segmenter's soft mask probabilities. We add one more term: the object's compliance (how much it sags) under its own weight, supported where it touches the ground plane, computed by a finite-element solve at every step. The claim is narrow and testable. For load-bearing man-made objects such as chairs, tables, brackets and stands, this single physical term removes ghost geometry and regrows members that segmentation eroded. It also resolves some depth ambiguities that pure image losses cannot. Crucially, it does so better than simply deleting disconnected pieces or adding a generic binarization penalty.

Core Claim

Among all 3D density fields that agree with one image, the ones that stand up under gravity on the visible support are a much smaller and more plausible set for furniture-like and mechanical objects. This has three consequences. First, a component with no load path to the ground makes the stiffness matrix nearly singular, so its compliance explodes and its gradient pushes it either to vanish or to connect. Second, an eroded leg leaves the seat hanging, and the compliance gradient is largest exactly along the missing path to the floor. This is the key difference from connected-component filtering, which can only delete the seat or leave it floating, never rebuild the leg. Third, along an ambiguous viewing ray, material placed over the support is cheaper than material placed as a far cantilever. This is a bias, not a truth, and we treat it as a hypothesis to measure, including where it hurts.

The objective over element densities $\rho \in [0,1]^N$ is

$$ \min_{\rho}\; \mathcal{L}_{\text{mask}}(R(\tilde\rho), m) + \alpha\,\mathcal{L}_{\text{depth}}(R_z(\tilde\rho), d) + \lambda\, \frac{f(\tilde\rho)^\top u}{C_0}, \quad K(\tilde\rho)\,u = f(\tilde\rho). $$

The terms are defined as follows:

  • $\tilde\rho$ is the filtered physical density.
  • $R$ and $R_z$ are differentiable volume renderings of silhouette opacity and expected depth.
  • $m$ is the segmenter's per-pixel foreground probability, kept soft rather than thresholded.
  • $d$ is a monocular depth estimate used as a weak prior.
  • $K(\tilde\rho)$ is the finite-element stiffness matrix with element modulus $\tilde\rho_e^{\,p}E_0$, the power-law penalty that makes half-dense material inefficient.
  • $f(\tilde\rho)$ is the self-weight load, proportional to density, pointing along the estimated gravity direction.
  • $u$ is the resulting displacement, with zero-displacement supports on voxels touching the fitted ground plane.
  • $C_0$ normalizes compliance by its value at initialization.
  • $\lambda$ trades physics against image evidence.

The compliance gradient comes from the standard adjoint, at the cost of one extra linear solve per iteration.

Research Program

Each ingredient comes from a specific prior work. The density representation and the power-law penalty come from density-based topology optimization. Topology Optimization of the Bracket Structure in the Acquisition, Pointing, and Tracking System Considering Displacement and Key Point Stress Constraints applies this interpolation (known as SIMP) to a real aerospace bracket under displacement and stress constraints. That work optimizes against loads only and needs manual CAD remodeling afterwards. We couple the same machinery to an image loss instead.

For the filter that produces $\tilde\rho$, we reuse the normalized field product from Normalized field product approach: A parameter-free density evaluation method for close-to-binary solutions in topology optimization with embedded length scale. It drives densities to nearly binary values, reporting grayness below 1%, without a tuned projection schedule, and it enforces a minimum member thickness. That thickness guard matters here, because otherwise the physics term could be satisfied by one-voxel threads and single-node hinges.

Initialization is uniform intermediate density inside the camera frustum. This matters because A Theory of Topological Derivatives for Inverse Rendering of Geometry shows that level-set inverse rendering can punch holes in 3D but explicitly cannot nucleate new disconnected material along a line of sight. A relaxed density field has no such restriction, so we can test whether it recovers components the level-set method cannot start.

The rendering side follows SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization in scale, working at $64^3$ grids with coarse-to-fine upsampling. We render densities by volume integration rather than sphere tracing, so that the image loss and the finite-element model act on the same variables.

Ground plane and gravity direction are estimated from the image with a standard plane fit on the monocular depth map. On synthetic data we also run with the true plane, to separate estimation error from the value of the prior.

The finite-element solve uses eight-node hexahedra on the density grid, with a void modulus floor of $10^{-6}E_0$ and a geometric multigrid preconditioned conjugate gradient on GPU. At $64^3$ this is roughly 260k elements. We expect the solve to cost the same order as the rendering pass, and we will report wall-clock time honestly.

Self-weight problems in topology optimization have a known pathology. Because load scales linearly with density but stiffness scales faster, low densities can look attractive. We therefore use a volume budget tied to the mask-consistent volume and a modified low-density interpolation. The sweep over $\lambda$ is part of the study, not a hidden hyperparameter.

Baselines

The naive reference is the same density optimization with $\lambda=0$ and a hard thresholded mask. This is the cascaded pipeline in its purest form: image fit plus depth prior, with no physics and no uncertainty in the mask. Every gain we report is measured against it first.

The strongest optimization-based competitor is SDFDiff, which reports single-view ShapeNet mean IoU of 0.6674, against 0.6464 for SoftRas and 0.6015 for NMR (SDFDiff). We run it with the same masks, depth prior, resolution and iteration budget. Beating it means higher IoU on the clean-mask chair and table categories, with the margin widening as mask noise grows.

The strongest learned competitor for topology is Topology Modification Networks. It deforms a sphere and prunes faces with high predicted error, and it reports chair Chamfer Distance 3.064 against 3.796 for AtlasNet-25 (Deep Mesh Reconstruction from Single RGB Images via Topology Modification Networks). Its output is an open mesh, so we evaluate it after Poisson closing, as that paper's own caveat implies. We also include the topology-regularized pipeline of Topology-Aware Single-Image 3D Shape Reconstruction, which adds learned genus and connectivity surrogates. We found no reported numerical genus or connectivity error for it on Lacuna, so we will re-run it and compute those errors ourselves.

Beating the learned methods on clean ShapeNet is not our claim, since they carry category priors we lack. The claim is that under controlled mask corruption our Chamfer and connectivity errors degrade more slowly than theirs. We will also check whether initializing our optimization from their output improves on it.

Two ablations isolate the key ingredient. The first replaces the compliance term with a differentiable connectivity penalty plus largest-connected-component cleanup. The second replaces it with an entropy binarization penalty, keeping the nFP filter. Each gets the same tuning budget as our $\lambda$ sweep: the same number of values searched on the same validation split.

Why not just…?

Why not just keep the largest connected component? This removes ghosts for free, and on over-inclusive masks it may match us. It cannot repair under-inclusive masks, because a seat cut off from its eroded legs is either deleted or left floating. Our erosion-only corruption experiment is designed to show this gap directly. If connected-component cleanup matches us there too, the mechanics term is not earning its cost and we will say so.

Why not just use a learned generative shape prior? Priors fill in plausible geometry. The closest robust example, ShapeR: Robust Conditional 3D Shape Generation from Casual Captures, reports a 2.7x Chamfer improvement on cluttered captures. However, it depends on posed video and SLAM points, so it does not apply to one unposed image. Learned single-image priors remain a real threat on in-distribution categories. We therefore test the prior and the physics as complements: learned-model initialization with and without the compliance term. We also report out-of-category mechanical parts, where learned priors are known to fill hollow geometry (SCADE: NeRFs from Space Carving with Ambiguity-Aware Depth Estimates notes this failure for depth priors).

Why not just produce better masks, or sample several? Accurate Localization of 3D Objects from RGB-D Data Using Segmentation Hypotheses raised B3DO 3D localization AP from 9.6% to 15.8% by sampling mask hypotheses. It needs sensor depth, and its recall stays capped by the upstream detector. The consistency checks of Efficiently Determining Silhouette Consistency have no pruning power with a single view. Still, better masks are a fair alternative, so we run everything with oracle ground-truth masks as well. Any gain that persists there comes from resolving depth ambiguity, not from mask repair.

Why not just a binarization penalty? Crisp densities alone do not choose where material goes along a ray. The entropy-plus-nFP ablation measures exactly how much of our gain comes from crispness rather than from load paths.

Proposed Experiments

The main testbed is ShapeNet chairs, tables and benches rendered on a ground plane. We apply controlled mask corruption at five severities: morphological erosion, dilation, shadow inclusion, and occluder cutouts. Metrics are IoU, Chamfer Distance, connectivity-number and genus-number error as in the topology-aware work, and a physical check: whether every solid voxel has a path to the ground, plus the finite-element compliance of the binarized result.

The first decisive check is the erosion curve. Our method must regrow severed legs, measured by per-leg recall, where connected-component cleanup cannot.

The second is oracle masks. A gain there would support the depth-disambiguation claim; no gain would restrict our contribution to mask repair.

For real images, we use Pix3D with masks from an off-the-shelf segmenter and estimated ground planes. For mechanical parts, we use ABC models (ABC: A Big CAD Model Dataset for Geometric Deep Learning) placed stably on a plane and rendered.

We deliberately include categories where the gravity assumption is wrong, such as hanging lamps and wall-mounted shelves. There we expect harm, and the size of that harm defines the prior's scope. A $\lambda$ sweep will map the trade-off between image fit and standing up. If the best $\lambda$ on validation gives no significant gain over the connectivity ablation on the corrupted benchmark, the central claim is falsified.