A Grasp Is a Caliper: Recovering Object Scale from Hand Contact in a Single Image

A single photograph cannot tell a toy mug held near the camera from a large mug held farther away. Both project to the same pixels, because perspective divides every coordinate by depth. Current hand-object reconstruction methods avoid this problem by assuming they already have the object's 3D model at its true size. Monocular depth methods instead resolve size using camera metadata, extra sensors, or assumptions about body height. We propose a third source of size information that is present in every grasping image: the hand itself. Adult hands vary in size within a narrow, known range. When fingers wrap around an object, the object has to be large enough for the fingers to touch it and small enough that the fingers do not pass through it. A grasp therefore works like a caliper. If the object is scaled too large, the hand penetrates it. If it is scaled too small, the fingertips float in empty space.

The core claim is that the balance between contact and non-penetration, which the hand-object literature uses only to clean up hand pose, can serve as a measurement of the object's absolute scale and depth. Recovering scale this way needs no CAD scale, no LiDAR, no inertial sensor, and no learned depth prior on the object. Where the measurement fails, the reason should be predictable from the grasp's geometry. A two-finger pinch at the tip constrains scale weakly. A power grasp with thumb opposing the other fingers constrains it strongly. We also borrow one method from differentiable quantum architecture search for the discrete step of deciding which hand regions are in contact. We run its quantum form only as an explicit falsification test, and we expect the classical version to be at least as good.

The Ambiguity, Stated Exactly

Let an object's shape be known only up to scale, as a mesh with vertices $u_i$ in a normalized frame, placed by rotation $R$ and translation $t$. Scaling the whole placed object about the camera center by a factor $s$ leaves its image exactly unchanged. The vertices become $s(Ru_i + t)$. This one-parameter family is the monocular scale ambiguity for the object.

The hand has the same ambiguity along its own viewing ray. Its absolute size, however, is pinned by a prior over the parameters of a parametric hand model (MANO, a standard low-dimensional model of hand shape and pose). With the hand fixed in metric space, we choose $s$ together with hand pose $\theta$ and hand shape $\beta$ by minimizing

$$E(s,\theta,\beta) = \lambda_c\, C(s,\theta,\beta;\, m) + \lambda_p\, P_\delta(s,\theta,\beta) + \lambda_\beta \|\beta\|^2 + E_{2D}(\theta,\beta).$$

The terms are defined as follows. $C$ is the distance from the hand regions marked in contact by a binary assignment $m$ to the object surface. $P_\delta$ penalizes penetration depth beyond a soft band of width $\delta$. $\|\beta\|^2$ keeps the hand within the population size prior. $E_{2D}$ is the reprojection error of hand keypoints.

The term that fixes scale is the opposition between contacts on opposite sides of the object. Thumb and fingers on opposite faces set a metric aperture that only one $s$ can satisfy. A single fingertip on one face satisfies contact for a range of $s$, and in that case we expect, and will measure, a flat valley in $E$.

Relation to Prior Work

The penalty terms come from existing work. We take the soft penetration band from ContactOpt, which showed that forbidding all penetration destabilizes optimization and that allowing about 2 mm of "skin squish" gives stable gradients. We take attraction-repulsion along surface normals from the virtual springs of CPF, and anatomical joint limits from its A-MANO model. Joint limits matter here because, as HO-3D_v3 and CPF both note, an unconstrained hand will contort its thumb to satisfy collision losses. That contortion would contaminate a scale estimate. Grasping Field encodes the same contact and non-overlap structure as paired signed distances, and could replace our mesh penalties.

All of these methods hold object scale fixed from a CAD model and optimize only the hand. We make object scale the unknown and treat the hand as the ruler.

Several papers motivate the gap. Metric3D v2 resolves metric depth by mapping images into a canonical camera, but it needs focal-length metadata. It also measures depth of visible surfaces, not the size of a partially occluded held object. LiDARTouch and Monocular Visual-Inertial Depth Estimation anchor scale with sensors we assume are absent. Single View Physical Distance Estimation using Human Pose is the closest in spirit: it uses human body height as a ruler. Its documented weakness under heterogeneous anthropometry, such as children versus adults, constrains our design. Hand-size uncertainty passes directly into object-scale error, so we must report it rather than hide it. Learning to Estimate 3D Hand Pose from Single RGB Images recovers hand pose only in a scale-normalized frame. It shows that hand size alone, without contact, leaves the object unconstrained.

Research Program

The pipeline has four stages. First, an image-based estimator predicts hand pose and a coarse, scale-normalized object shape and pose. The object shape comes either from the dataset's CAD model rescaled to unit size, or from a category-level template when no CAD model is available. Second, a contact predictor proposes a soft contact map on the hand. We use ContactOpt's DeepContact, trained on ContactPose, which supplies real thermal-imaging contact data. Third, we jointly optimize $(s,\theta,\beta)$ under $E$. Because $E$ is nearly one-dimensional in $s$ once the hand is fixed, we sweep $s$ over a grid at each outer iteration rather than trusting gradient descent alone. This sweep exposes multimodality, for example when a mug handle and its body admit different contact readings. Fourth, we report a scale estimate together with a confidence interval, taken from the width of the low-energy valley in $s$.

The discrete contact assignment $m$ is the step where the architecture-search literature supplies a method. Differentiable quantum architecture search relaxes discrete gate choices into a probability-weighted mixture and trains the mixture weights jointly with continuous parameters using score-function gradients. We apply the same relaxation classically: each of roughly 16 hand regions gets a Bernoulli contact probability, optimized jointly with $s$.

As a bounded side study, we also write the assignment-plus-binned-scale problem as a small quadratic binary objective. This follows the way Differentiable Quantum Architecture Search For Job Shop Scheduling Problem cast scheduling overlaps. We search circuits for it in simulation. With 16 binary contacts plus a few scale bits, the problem fits in the 2–8 qubit regime where these methods have been shown to work. Exhaustive classical enumeration (about $10^5$ to $10^6$ states) gives the exact answer, so this arm tests whether searched circuits reach the exact optimum. It makes no claim of advantage.

Baselines

The naive reference fixes object scale at the category mean size from the training split, then runs the same contact optimization over hand pose only. Any method that cannot beat category priors has learned nothing from the grasp.

The strongest existing competitor for scale is metric depth from Metric3D v2. We run it zero-shot on the same frames with the dataset intrinsics supplied, which favors it. We fit the object's scale and depth to the predicted depth over the visible object mask by least squares. For the strongest variant, we then also apply our contact term. Metric3D v2 reports zero-shot Abs Rel of 0.047 on NYUv2 and 0.044 on KITTI. No reported result was found on Lacuna for Metric3D v2 on HO-3D or on hand-held objects. Close-range, heavily occluded objects are a different regime, so its accuracy there is an empirical question we will answer. Beating it means lower median object-scale error on HO-3D_v3 at matched intrinsics. Matching it without intrinsics also counts as a meaningful result.

For hand and contact quality, the strongest baseline is ContactOpt, which reports MPJPE falling from 57.7 mm to 48.1 mm on HO-3D. We run it in two configurations: with oracle CAD scale, as an upper bound, and with the naive category-mean scale, as the realistic deployment setting. Both use the same DeepContact predictions, iteration counts, and GPU. For penetration we cite CPF, which reports a 3.71 mm reduction in penetration depth and a 9.34 cm³ reduction in intersection volume on FHB. We will run CPF's released optimization stages under the same scale-withheld protocol. Success means coming within a small margin of oracle-scale ContactOpt on MPJPE and penetration while clearly beating ContactOpt with category-mean scale.

The ablations isolate the caliper mechanism. One removes the penetration term, leaving contact only. One removes the contact term, leaving penetration only. One fixes hand shape $\beta$ to the mean, which tests how much the hand-size prior carries. One masks the opposing contacts so that only fingers on a single side remain. If opposition is the mechanism, the single-side ablation should collapse scale accuracy while leaving hand pose error nearly unchanged.

Why not just…?

Why not just read object size off a monocular metric depth model? Metric3D v2 is the real competitor and is run as one. Its published accuracy is on scene-scale benchmarks, and it needs focal length. A held object is small, partly hidden by the hand, and only centimeters from the hand surface. A few percent of relative depth error on the hand-object region could therefore be a large fraction of the grasp aperture. The experiment settles this: if Metric3D v2 with intrinsics matches our scale error, our contribution reduces to the intrinsics-free setting and to the combined variant. We will say so.

Why not just add a cheap sensor? LiDARTouch shows that four LiDAR beams eliminate catastrophic depth errors on KITTI, and visual-inertial anchoring does similar work at over 50 FPS. We do not compete with sensor rigs. The target is the very large body of existing RGB-only hand-object video, such as egocentric footage and internet clips, which has no sensor channel and often no reliable metadata.

Why not just use classical optimization and skip anything quantum? This alternative is a real threat. We accept it for the main method, which is entirely classical and borrows only the mixture relaxation from the architecture-search work. The quantum arm is retained as a falsification test of whether searched circuits recover the exact optimum of a small, real geometric assignment problem. Earlier QAS demonstrations used chemistry, MAX-CUT, and scheduling. Quantum circuit architecture search for variational quantum algorithms reports energy -0.96 Ha against an exact -1.136 Ha on hardware, so a gap to the exact optimum is the expected outcome. If the quantum arm underperforms exhaustive enumeration, we report that negative result. Nothing in the main claim depends on it.

Why not just rely on the hand-size prior without contact? Hand size alone places the hand in metric space but leaves the object free along its ray, as the scale-normalized lifting in Learning to Estimate 3D Hand Pose from Single RGB Images illustrates. The no-contact ablation measures exactly how much contact adds.

Proposed Experiments

The first experiment is a controlled identifiability test on ObMan, where true scale is known exactly. We rescale objects by factors from 0.7 to 1.4 and check whether the energy minimum returns to 1.0. We break results down by grasp type and by the number of opposing contacts. The expected pattern, sharp minima for power grasps and flat valleys for tip pinches, is the test of whether the caliper idea is real.

The main evaluation uses HO-3D_v3 and FHB with the CAD scale withheld from the method. Metrics are relative object-scale error, object translation error along depth, hand MPJPE, penetration depth, solid intersection volume, and contact ratio. We also report how well the confidence interval is calibrated against realized error.

The robustness experiment injects hand-size mismatch by drawing hand shape from tails of the prior, standing in for children's hands. It quantifies how hand-size error propagates into object-scale error. This is the same failure mode the human-pose distance paper faces with body height.

The decisive outcome has two parts. First, on scale-withheld HO-3D_v3, contact-based scale must beat the category-mean baseline. Second, without intrinsics, it must reach object-scale error comparable to Metric3D v2 with intrinsics, while hand MPJPE stays close to oracle-scale ContactOpt. If the single-side contact ablation does not degrade scale accuracy, the caliper explanation is wrong and the gains come from something else. That result would also be worth reporting.