Many methods that rebuild a 3D mesh from one photograph refine the mesh in a loop. They project each vertex onto the image and read the image features at the pixel where it lands. Those features are then used to decide how to move the vertex. The step has a blind spot. Every point along the camera ray through that pixel reads exactly the same feature. So a vertex that is 30 centimetres too far from the camera receives the same evidence as one at the correct depth. The loop can drag vertices sideways onto image edges, but it gets no direct signal about whether they sit at the right distance. It also gets no signal about whether they are hidden behind something else. This is the single-image depth ambiguity showing up inside the refinement step itself.
We propose to give each vertex evidence about its depth as well as its position in the image. An off-the-shelf monocular depth and surface-normal predictor supplies a second input alongside the RGB features. At every refinement step, that relative depth map is fitted to the current mesh with a per-object scale and shift. Each vertex then gets two cues. The first is a depth residual: how far its current depth is from what the fitted map says at its pixel. The second is a visibility flag: whether the fitted map says something sits in front of it. A small attention module compares a few candidate depths along the vertex's ray and predicts a move along the ray. The claim is that this gives mesh-refinement loops the missing depth signal. Gains should therefore land mainly in the camera-axis component of the error and on occluded vertices, which is where today's methods fail.
Core Claim
Vertex-aligned refinement, as done in Mesh R-CNN and PyMAF, cannot correct depth error from the features it samples. For fixed image coordinates $(u,v)$, the sampled feature does not change with the vertex's depth $z$. Any depth correction must therefore come from learned priors, not from image evidence.
We test a narrow hypothesis. Suppose each vertex is given an explicit, aligned depth residual and a visibility cue, both computed from an external depth predictor. Then per-axis error along the optical axis should fall clearly, and the in-plane error should stay roughly unchanged. The residual and visibility cue should matter more than simply having the depth map available.
The idea differs from its closest relatives in specific ways:
- DiMeR shows that feeding RGB into geometry decoders lets painted texture fool the geometry. It routes predicted normals into a triplane decoder instead, cutting Chamfer distance on Google Scanned Objects by 31.7%. It never projects vertices back for local refinement, and it uses normals as input, not as a comparison against the current mesh.
- CramNet samples points along camera rays and cross-attends to radar features to place detections at the right depth. It outputs boxes, and its depth evidence is a metric sensor.
- Here the rays pass through evolving mesh vertices, and the depth evidence is a relative, scale-free prediction. That prediction must be re-fitted to the mesh at every step.
Research Program
We start from two existing refinement architectures and change only the per-vertex evidence. The first is PyMAF-style parametric body refinement on SMPL, the standard parametric body mesh model. The second is Mesh R-CNN-style GCN deformation for arbitrary object topology. We add hand meshes following Pixel-Aligned Non-parametric Hand Mesh Reconstruction.
A frozen monocular estimator produces a relative depth map $\hat d$ and a normal map $\hat n$. DiMeR shows such normal predictors, for example Lotus and StableNormal, are usable geometry inputs. At refinement step $t$, the mesh has vertices with camera-frame depth $z_i$ projecting to $(u_i,v_i)$. We fit a scale $s$ and shift $b$ with a robust (Huber) least-squares fit over the vertices that the mesh's own z-buffer marks as front-facing:
$$ (s,b)=\arg\min_{s,b}\sum_{i\in V_t}\rho\big(z_i - s\,\hat d(u_i,v_i) - b\big), $$
Here $V_t$ is the set of currently self-visible vertices and $\rho$ is the Huber loss. From the fit we compute each vertex's aligned residual $r_i = z_i - s\hat d(u_i,v_i) - b$.
A strongly positive residual means one of two things: the vertex is behind the surface the depth map sees, so it is occluded, or it is too far away. Which one applies is decided by the mesh's own z-buffer. A vertex the mesh itself marks as front-facing should have near-zero residual. A vertex the mesh marks as self-hidden is expected to have a positive one.
We then form $K$ depth hypotheses $z_i+\delta_k$ along the vertex's ray, with $K\approx 8$ and a fixed set of offsets $\delta_k$. Each hypothesis is described by:
- its residual,
- the agreement between $\hat n$ and the mesh normal at that vertex,
- and the self-visibility flag.
The vertex's RGB feature acts as the query in a small cross-attention over these hypotheses, in the spirit of CramNet. Its output is a scalar move along the ray, which is added to the usual in-plane update. The added module is cheap because it reads only one extra pixel lookup per map per vertex.
The analysis is as important as the method. Every result is reported with error split into the camera-axis component and the image-plane component. We also report error on vertices that the ground-truth mesh marks as occluded, separately from visible ones. Without this split, an averaged metric like PA-MPJPE can hide exactly the effect we claim. PA-MPJPE is mean per-joint position error (MPJPE) after a Procrustes alignment that removes global scale and rotation.
Baselines
The naive reference point is the unmodified refinement network with identical training data and schedule: PyMAF for bodies, Mesh R-CNN for Pix3D objects, and the pixel-aligned hand network for FreiHAND. The numbers to reproduce and then beat are:
- PyMAF's 92.8 mm MPJPE on 3DPW, against 96.9 mm for its SPIN baseline, as reported in PyMAF.
- 5.9 mm PA-MPJPE on FreiHAND from Pixel-Aligned Non-parametric Hand Mesh Reconstruction.
- For Pix3D, the file gives only that Mesh R-CNN reports a 20% mesh-AP gain over its baselines on bookcases. We found no absolute Chamfer or F1 value for it on Lacuna, so we will rerun it ourselves.
The strongest existing single-image mesh method in the file is PointHMR. It samples features at predicted vertex heatmap locations and reports 48.3 mm MPJPE on Human3.6M, against 51.2 mm for Mesh Graphormer and 52.2 mm for FastMETRO. It also reports 6.1 mm PA-MPJPE on FreiHAND. Because it is itself a sampling method, we also graft our module onto it. Beating it means lowering Human3.6M MPJPE below 48.3 mm, with the reduction concentrated in the camera-axis component. A gain that appears only in the image plane would not support the claim.
Two further baselines use the same depth and normal maps and so isolate the method from the extra input:
- Channel concatenation. The maps are stacked as extra input channels to the backbone.
- Naive vertex concatenation. The raw values $\hat d(u_i,v_i)$ and $\hat n(u_i,v_i)$, plus the vertex's $z_i$, are appended to each vertex feature, with no scale-shift fit, residual, visibility flag or hypotheses.
A post-hoc optimization baseline takes the refined mesh and adjusts it to minimize a scale-shift-invariant loss against its rendered depth.
Ablations remove one ingredient at a time: the alignment fit, the visibility flag, the ray hypotheses (predicting a move from the residual alone), and the normal-agreement term.
All variants share the following:
- the same backbone and training data;
- the same frozen depth and normal predictor;
- the same number of refinement steps;
- an equal hyperparameter-search budget of a fixed number of trials per method;
- three seeds each.
Why not just…?
Why not just feed the depth and normal maps as extra input channels? This is the most likely objection, and it is a real threat. The backbone would then encode depth at pixel $(u,v)$ into the sampled feature. But the vertex's own depth would still not enter that feature, so the network must learn to compare the two implicitly. Because the monocular depth is only relative, it must also learn the per-image scale. The channel-concatenation baseline tests this directly. If it matches our camera-axis gains, then the explicit residual adds nothing, and we will report that outcome.
Why not just concatenate the sampled depth value with the vertex's depth and let the network sort it out? This is Alternative 2 in its strongest form, and it is also a real threat. SGFormer reports that adaptive fusion beats naive concatenation by 2% to 10% under noise and occlusion, but that evidence comes from LiDAR panoptic segmentation, not mesh refinement (SGFormer). Our argument for doing better is that the scale and shift change with every image and every refinement step. The needed information is therefore a relation between vertices, not a per-vertex value. The naive-vertex-concatenation baseline and the alignment-free ablation will settle whether this matters. If the explicit fit adds less than about 1 mm on camera-axis error, we would drop the fit and keep the simpler design.
Why not just fit the mesh to the predicted depth by test-time optimization? Post-hoc optimization is the classic answer and needs no training. Its weakness is that it treats every vertex as visible. In practice it pulls occluded vertices, such as the back of an arm or a chair leg behind the seat, onto the occluder's front surface. This is the occlusion failure listed for projective sampling in the file. The occluded-vertex split is the test: we expect optimization to match us on visible vertices and lose on occluded ones. If it does not lose there, the learned module is unnecessary.
Why not just use video or a stronger backbone? Glimpse reports 39.3 mm MPJPE on Human3.6M from single frames with structural priors. It beats several video models, which suggests single-frame depth evidence is far from exhausted. We will also run a DINOv2 backbone variant of the naive refinement network. A backbone swap improves features but leaves the sampled feature unchanged along the ray, so we predict its gains fall mostly in the image plane.
Proposed Experiments
Bodies. We train on standard mixed human data and evaluate on 3DPW and Human3.6M. We report MPJPE, PA-MPJPE and vertex error, each split into camera-axis and image-plane components. We also report occluded-vertex error, using SMPL self-occlusion from ground truth plus synthetic occluder patches pasted onto test images at controlled coverage.
Hands. We use FreiHAND with PA-MPJPE and vertex error, and non-aligned root-relative error, since Procrustes alignment hides depth scale.
Objects. Pix3D is the arbitrary-topology test, since Mesh R-CNN's cubified meshes have no body prior to lean on. We report Chamfer distance, F1 and mesh AP, with depth-axis error of surface samples.
There are three decisive checks.
- Where the gain lands. The improvement over the naive-vertex-concatenation baseline should appear in camera-axis and occluded-vertex error and be near zero in the image plane. A uniform gain would suggest we have just added capacity.
- Dependence on depth-map quality. We swap the frozen depth predictor for one with corrupted depth, adding noise and blurring. Camera-axis gains should shrink in proportion, while in-plane accuracy should stay intact. DiMeR notes its accuracy is bounded by its upstream normal model, and this check measures the same dependence for us.
- Where Mesh R-CNN's depth drift is worst. On Pix3D, the Mesh R-CNN summary notes it struggles with depth and scale along the ray. We bin test instances by that per-instance depth error. Our gains should grow with the bin, since the module targets exactly this failure.
Failing the first check falsifies the core claim, regardless of headline MPJPE.
References
- Mesh R-CNN
- PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop
- DiMeR: Disentangled Mesh Reconstruction Model with Normal-only Geometry Training
- CramNet: Camera-Radar Fusion with Ray-Constrained Cross-Attention for Robust 3D Object Detection
- Pixel-Aligned Non-parametric Hand Mesh Reconstruction
- Sampling is Matter: Point-Guided 3D Human Mesh Reconstruction
- SGFormer: Semantic-Geometry Fusion Transformer for Multi-modal 3D Panoptic Segmentation
- Glimpse: Geometry Learning of Multi-scale Structural Priors for 3D Pose Estimation