A single photograph cannot tell a toy laptop close to the camera from a full-size laptop farther away. Every part of both objects projects to the same pixels. This is the monocular scale ambiguity. For articulated objects, meaning things with moving parts joined by hinges or sliders (laptops, drawers, scissors, robot arms, hands), a second ambiguity sits on top of it. Whether a lid is tilted toward the camera or away from it can produce nearly the same silhouette, so the joint angle is uncertain as well. Current single-image methods handle the first ambiguity by assuming a typical object size, and they refine the second with slow render-and-compare loops. Both approaches break when an instance is unusually sized or when the kinematic chain is long.
This proposal starts from a geometric observation. For an articulated object seen in one image, what the image leaves undetermined is small: one global scale factor, plus a few near-degenerate joint states. It does not take a dense depth map to remove a handful of unknowns. A few metric depth points from a low-cost sensor should be enough, such as a few-beam LiDAR, a phone time-of-flight sensor, or visual-inertial odometry landmarks, provided each point is assigned to the right rigid part and fed through the object's kinematic model. The network would not treat the points as another image channel to be learned from. We propose an RGB network that predicts a scale-free articulated reconstruction, followed by a small robust solver that fuses the sparse points at the level of kinematic parameters (scale and joint states) rather than pixels.
Core Claim
Under perspective projection, scaling the entire scene about the camera center by any factor $s>0$ leaves the image unchanged. This holds for any articulated configuration, because scaling preserves joint angles. An RGB network can therefore at best recover the object up to $s$. Let the network predict camera rotation $R$, joint states $\hat\theta$, a normalized shape, and a unit-scale translation. From these one can render, for any pixel $k$ assigned to part $j(k)$, a scale-free depth $\hat z_k(\theta)$. Given $K$ metric depth measurements $z_k$, we solve
$$ \min_{s,\,\theta}\; \sum_{k=1}^{K} \rho\big(z_k - s\,\hat z_k(\theta)\big) \;+\; \lambda \,\lVert \theta - \hat\theta \rVert^2 , $$
In this objective, $\rho$ is a robust penalty that downweights outlier points, and $\lambda$ controls how far the joint states $\theta$ may move from the network's prediction $\hat\theta$. For fixed $\theta$, the optimal $s$ is a weighted ratio in closed form. The joint states then need only a few Gauss–Newton steps, because each hinge or slider has a single degree of freedom.
The claim is that this kinematics-aware sparse anchoring recovers metric part poses and joint states with far fewer depth points than two alternatives. The first alternative is feeding sparse depth into the network. The second is aligning a monocular depth map to the points. The gap should be largest on long kinematic chains, where points on distal links correct accumulated drift, and on instances whose size departs from the category average.
The contribution is an operating point that has not been tested. Prior work anchors scale with sparse points only for rigid scenes, and it estimates articulation from RGB only with size priors. We connect them through the one quantity the image cannot see.
Relation to Prior Work
Three lines of work motivate the gap. R^2-Art estimates category-level articulation from one RGB image. It derives depth from the ratio of a known category template size to the object's pixel area and then refines with a three-stage differentiable rendering cascade. Its authors report error accumulation on long chains. Its size assumption is exactly what fails when instances are rescaled. Single View Physical Distance Estimation using Human Pose uses a statistical human height as a calibration rod and inherits the same intra-class variance problem. Learning to Estimate 3D Hand Pose from Single RGB Images lifts 2D keypoints to a normalized canonical hand and leaves metric size unresolved. We keep their decomposition into a normalized pose and a separate scale, but we replace the prior on scale with a measurement.
Two lines supply the method. LiDARTouch shows that fewer than 100 LiDAR points per frame fix metric scale for a whole driving scene. It fuses them early into depth-completion networks. It models no articulation, and its points are scene-level rather than assigned to parts. The part-assignment and kinematic-residual design comes from Category-Level Articulated Object Pose Estimation and OMAD. ANCSH uses predicted joint axes as "glue" that keeps parts attached during optimization. OMAD separates a low-rank shape code from joint states and fits sparse keypoints through a kinematic deformation function. Both consume full depth point clouds. We ask whether the same kinematic glue works when depth is reduced to a few points and appearance must supply everything else. That is also the RGB-fusion direction ANCSH names as future work. The RGB backbone follows R^2-Art's joint-centric parameterization, where the parent pose plus joint state determines the child pose.
Two papers constrain the design. PointFusion found that batch normalization harms absolute 3D coordinate regression. This is one reason we keep metric magnitudes out of the network and inside the solver. Mitigating Perspective Distortion-Induced Shape Ambiguity in Image Crops shows that off-center crops confound shape with perspective. It reports a 19% reduction in articulation angle error on ARCTIC from intrinsics-aware positional encoding. We adopt that encoding in the crop-based RGB network so that the residual ambiguity really is only scale and joint sign.
ART reconstructs articulated objects feed-forward from calibrated multi-view, multi-state images. It is complementary rather than competing, because it avoids monocular ambiguity by requiring more views.
Research Program
The system has three pieces. First, a single-image network, a DINOv2 backbone with intrinsics-aware crop encoding, predicts per-pixel part labels, part-level normalized coordinates, camera rotation, joint states, and translation up to scale. Second, sparse metric points are projected into the image and assigned to parts using the predicted labels, with the assignment soft-weighted by label confidence. Third, the robust solver above estimates $s$ and refines $\theta$.
We will analyze when joint states are identifiable from $K$ points. A single point on each of two parts fixes scale and constrains the relative depth of the child part, which for a revolute joint removes the toward-or-away sign ambiguity. We will measure how many points per link are needed as chain length grows, using the 7-link robot arm as the stress case. A training variant backpropagates through the solver, with the solver's closed-form scale being differentiable. This tests whether the network learns to put its uncertainty where the points can resolve it.
Baselines
The naive reference is R^2-Art's own scale mechanism: category template size divided by pixel area, with no depth points at all. It is the right floor because it is what single-RGB articulation currently uses. On ArtImage, R^2-Art reports laptop ADD of 37.5mm (versus 49.2mm for the best RGB-D baseline it compared against) and drawer ADD reduced from 180.5mm to 105.6mm across its cascade. On RobotArm it reports average ADD of 46mm with 68% AUC. We will reproduce R^2-Art under our splits and report both its published and reproduced numbers.
The strongest modular competitor uses Metric3D v2 to produce metric depth, which it reports at zero-shot Abs Rel 0.047 on NYUv2. That depth is back-projected and fed to OMAD, which reports 3.3° laptop joint angle error on ArtImage from true depth. In a stronger variant, Metric3D's depth is first least-squares aligned to the same sparse points our method receives. This variant gets the identical sensor information, so it isolates whether kinematic, part-aware fusion matters beyond scale alignment of a dense map.
The early-fusion competitor follows LiDARTouch. We give our own backbone the sparse depth as an extra masked input channel with a masked depth loss. It is trained on the same data with the same compute and the same hyperparameter search budget.
Two ablations isolate the key ingredient. One solves only for $s$ and leaves $\theta$ fixed. The other drops part assignment and treats the object as one rigid body. All methods see the same images, intrinsics, and point samples. Anchor counts are swept from 1 to 64 points on the object, with realistic noise and small calibration offsets.
Beating the baselines means lower metric ADD and joint-state error at every anchor count up to about 16 points, with a gap that grows on RobotArm and on rescaled instances. We would also want to match early fusion when points are plentiful.
Why not just…?
Why not just run Metric3D v2 and hand the depth to OMAD? Metric3D v2 needs focal length to be metric, and it smooths thin structures such as scissor blades and laptop lids. Towards Ambiguity-Free Spatial Foundation Model reports that larger depth models commit confidently to wrong layers on glass and reflective surfaces, which appliance doors often have. The version with sparse-point alignment is a genuine threat, though. If it matches our solver, the kinematic model adds nothing. The anchor-count sweep settles this directly. We expect it to fix scale well but leave joint states biased by boundary errors that no global alignment can correct.
Why not just assume the category's average size? Size priors fail exactly when instances differ from the mean. That is the known weakness of the pedestrian-height method, and R^2-Art's template ratio shares it. Our rescaled-instance test multiplies ArtImage and SAPIEN object sizes by factors from 0.5 to 2. A prior-based method should show metric error proportional to the rescaling, while an anchored method should be flat.
Why not just feed the sparse depth into the network, as LiDARTouch does? This is the most serious alternative. LiDARTouch shows it works for scenes, reaching 2.504m RMSE on KITTI with 4 beams against 2.112m for 64-beam supervision. Our hypothesis is that with only a few points on the object, a network cannot reliably learn to route each point to the correct link and propagate it through the chain, while an explicit solver does this by construction. If early fusion matches us at low anchor counts and under anchor patterns unseen during training, we would recast the contribution as the solver serving as a training target for early fusion, rather than as a replacement for it.
Proposed Experiments
Training uses SAPIEN and ArtImage renderings. For those, and for ARCTIC, we simulate sparse points from ground-truth meshes, using scan-line patterns that mimic few-beam LiDAR and random subsets that mimic visual-inertial landmarks. Evaluation reports per-part ADD in millimetres, joint angle and translation error, and scale error as a percentage. It covers ArtImage (laptop, eyeglasses, dishwasher, scissors, drawer), RobotArm for long chains, and ARCTIC for hand-held articulated objects.
On ARCTIC, ArcticNet reports roughly 20–23mm hand MPJPE. No reported absolute articulation-angle number was found on Lacuna for ArcticNet or for the KPE variant, only the 19% relative gain. We will therefore rerun both.
The three checks that would decide the claim are as follows. The first is error versus anchor count, where our curve should reach its plateau with a handful of points while dense alignment and early fusion need many more. The second is error versus instance rescaling, where anchored methods should stay flat and prior-based methods should grow linearly. The third is per-link error along the RobotArm chain, where distal-link error should drop the most once points fall on those links.
A negative result on the third check, with scale fixed but chain drift untouched, would show that sparse anchors solve scale but not degree-of-freedom complexity. That would narrow the claim accordingly.
References
- R^2-Art: Category-Level Articulation Pose Estimation from Single RGB Image via Cascade Render Strategy
- Single View Physical Distance Estimation using Human Pose
- Learning to Estimate 3D Hand Pose from Single RGB Images
- LiDARTouch: Monocular metric depth estimation with a few-beam LiDAR
- Category-Level Articulated Object Pose Estimation
- OMAD: Object Model with Articulated Deformations for Pose Estimation and Retrieval
- PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation
- Mitigating Perspective Distortion-Induced Shape Ambiguity in Image Crops
- ART: Articulated Reconstruction Transformer
- Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation
- Multi-Hypothesis Spatial Foundation Model: Rethinking and Decoupling Depth Ambiguity via Laplacian Visual Prompting