A single color camera cannot tell a small nearby mug from a large distant one. This scale ambiguity is the main reason RGB-only six-degree-of-freedom grasping is still unreliable. Six-degree-of-freedom (6-DoF) grasping means choosing the full 3D position and 3D orientation of the gripper, not just a top-down angle. Yet nearly every tabletop grasping setup already holds the missing information and does not use it: a calibrated camera looking at a known support plane. We propose to treat that plane as a metric ruler. A frozen pre-trained image encoder does not try to predict depth in meters, which such encoders do poorly. Instead it predicts, for every object pixel, a purely two-dimensional quantity: where in the image the point's vertical projection onto the table would land, which we call its "foot." Given that foot point and the known plane, metric 3D follows from the line through the camera and the pixel and a vertical drop to the table, with no learned scale at all. A light head then places a gripper keypoint model in this metric reconstruction to produce full 6-DoF grasps.
The three directions play different roles here. The planar-grasping tradition contributes the calibrated, gravity-aligned table frame. We keep that frame as a measuring device while discarding its restriction to top-down actions. Pre-trained visual representations contribute the per-pixel features. We ask them only for scale-free image-plane predictions, which matches what they are good at. Pose estimation contributes the core move of structure-based localization: predict correspondences, then let geometry recover the pose rather than regressing the pose directly.
Core Claim
Frozen 2D features plus a support-plane constraint should recover millimeter-to-centimeter metric geometry of objects on a table from one oblique RGB view. The result should be good enough for unconstrained 6-DoF grasping. This should hold even on transparent and reflective objects where depth sensors fail. It should also clearly beat two alternatives: regressing metric depth or grasp parameters directly, and reconstructing up-to-scale geometry and then fixing one global scale with the table.
What is new is the parameterization. The network never outputs a length. Every metric quantity comes from calibration plus a per-pixel foot correspondence, in the same way scene coordinate regression gets precision from a geometric solver rather than from the network's weights. The underlying idea is classical single-view measurement against a reference plane. As far as the file shows, it has not been combined with learned features for grasping.
Relation to Prior Work
The closest single-view system is MonoGraspNet. It detects 2D grasp keypoints, estimates surface normals, and uses a Swin-Transformer on image crops to regress keypoint depth, gripper width and tilt directly. Its own limitations section reports that small pixel errors become large metric translation errors and that it enforces no global geometric consistency. Our proposal replaces exactly the depth-regression step with plane-anchored lifting.
MG-Grasp takes the opposite route. It runs the pre-trained 3D model MASt3R over five to six views, recovers metric scale by triangulating across views with known camera poses, and refines the result with bundle adjustment. It works well but needs multiple views and about 2.86 s. We replace triangulation across views with a geometric constraint inside a single view.
These two works motivate the proposal by locating the gap, which is metric scale from RGB. They also constrain it: MG-Grasp reports failures on reflective and near-spherical objects where dense correspondences break, and we test exactly those objects.
The method borrows from pose estimation. Scene Coordinate Regression Forests and Bui et al. show that predicting per-pixel correspondences with confidence and then solving geometrically closes the precision gap that PoseNet-style direct regression cannot. PoseNet's median errors are about 0.5 m and 5° on 7 Scenes. Our foot-point map is a scene coordinate map whose "scene" is the calibrated table, and we reuse Bui et al.'s idea of per-pixel trust scores as weights in the final fit.
6-DoF Object Pose from Semantic Keypoints warns that perspective-n-point solvers degenerate when correspondences are coplanar. That warning matters because a parallel-jaw gripper's keypoints are coplanar. So we lift keypoints to 3D first and fit the gripper model by weighted rigid alignment, which avoids that degeneracy.
EPro-PnP shows how to train correspondences end to end through a probabilistic pose layer. We treat it as an optional training refinement, not the core mechanism, because its sampling cost buys little once lifting is closed-form.
On the representation side, Parisi et al. and Real-World Robot Learning with Masked Visual Pre-training establish that frozen encoders support data-efficient control. CLAP uses frozen DINOv2 for geometric features, but feeds it depth maps from a sensor. The file's account of the field notes that these encoders are trained to ignore metric scale. We take that as a design rule rather than a problem to fight.
Finally, Equivariant Transporter Network and GEM get sample efficiency from symmetry under rotations in the table plane, but cannot express out-of-plane grasps. Our gravity-aligned table frame keeps that symmetry about the table normal, so it can be preserved for the yaw part of the grasp while pitch and roll stay free.
Research Program
The system has three stages.
The first stage predicts foot points. A frozen DINOv2 encoder, the same family CLAP uses, feeds a small convolutional or attention head. For each pixel $u$ the head predicts a 2D image offset $\delta(u)$ to its foot point $f(u)=u+\delta(u)$ and a confidence $c(u)$. A foot point can be occluded, for example under an overhang or a stacked object. It remains well defined as an image location, and the head must infer it from shape cues.
The second stage lifts points to 3D in closed form. Let $K$ be the camera intrinsics and $(R,t)$ the camera-to-table pose, known from the calibration that planar setups already perform. The 3D table point under $u$ is the intersection of $f(u)$'s camera ray with the plane $z=0$, which gives its table coordinates $(x,y)$. The point itself lies on $u$'s ray at the location whose vertical projection is $(x,y)$, and this fixes its height $z$. All metric content comes from $K$, $R$ and $t$.
Lifting is ill-conditioned when $u$ is close to the image location where vertical lines converge, which for a top-down camera is near the image center. The height error therefore grows as the camera looks more nearly straight down. We will measure this dependence on camera elevation explicitly rather than hide it, since it tells practitioners where to mount the camera.
The third stage fits the grasp. A grasp head reads the same frozen features and proposes, per candidate, image locations of gripper keypoints: two finger contacts, an approach point and a wrist point. These keypoints are lifted through stages one and two. The gripper's known metric keypoint model $G$ is then aligned to them by confidence-weighted rigid alignment:
$$ T^\ast=\arg\min_{T\in SE(3)}\sum_k c_k\,\|T\,G_k-\hat X_k\|^2 , $$
where $\hat X_k$ is the lifted 3D estimate of keypoint $k$, $c_k$ its confidence, $G_k$ its position in the gripper's own frame, and $T$ the gripper pose. This alignment is differentiable, so the keypoint head can be trained through it. Grasp width comes from the lifted contact distance, and the lifted point cloud supports a simple collision check against the gripper mesh. That check addresses a limitation MonoGraspNet names.
Training supervision for foot points comes for free wherever sensor depth is valid. Any opaque pixel with a measured 3D point gives its foot point by vertical projection and reprojection. This lets us train on GraspNet-1Billion and test on transparent objects without transparent-object depth labels. If MonoGraspNet's multi-view dataset with its roughly 20 million grasp labels can be obtained, we will add it.
Baselines
The naive reference uses the same frozen DINOv2 encoder and the same grasp head and alignment step, but the head regresses metric depth per pixel instead of foot offsets. This is the "just ask the features for depth" design. It shares every component except the parameterization, so the gap between it and our method is the cleanest measure of the core claim.
The key ablation feeds the predicted foot geometry into the same pipeline but replaces it with a predicted height above the table: a metric scalar combined with the plane. This isolates whether the gain comes from using the plane at all or from predicting only scale-free 2D quantities. Two further ablations unfreeze the encoder, which tests whether freezing costs accuracy, and drop the confidence weights.
The strongest existing single-view competitor is MonoGraspNet. It is the published RGB-only 6-DoF method designed for photometrically hard objects, and the file reports that it reaches 80% success on a transparent cocktail glass. On the same glass, Contact-GraspNet with sensor depth reaches 6.7%. MonoGraspNet also drops from 83.9% to 72.2% when moving from familiar to novel objects. We will retrain it on our training split with the same number of GPU-hours and the same hyperparameter search budget as our head, and evaluate both on identical frames. Beating it means higher success on novel transparent and reflective objects and a smaller drop from familiar to novel, since that drop is where we expect scale errors to bite. No reported result for MonoGraspNet on GraspNet-1Billion was found on Lacuna, so that comparison will be produced fresh.
MG-Grasp serves as an upper reference rather than a direct competitor, because it uses five to six views. The file reports it exceeds the previous best RGB-only method VG-Grasp by +12.18/+14.33 AP on GraspNet-1Billion's two camera splits, and reaches 87.5% real-world clutter-clearing success. No absolute AP value appears in the file. We will rerun its released configuration on the same splits. A single-view method coming within a few AP of it at a fraction of its 2.86 s latency would be a meaningful result. Sensor-depth grasping on the same scenes is the RGB-D reference.
Why not just…?
Why not just take an up-to-scale reconstruction and fix the scale using the table? This is the most serious threat. One can run a pre-trained geometry model such as MASt3R, the reconstruction module inside MG-Grasp, on a single view, fit the table plane in its output, and pick the scale and offset that place that plane at the calibrated table. It reuses the same plane information with no new training. The difference is that a global correction can fix only one scale and one offset, while our method anchors each pixel separately. If the pre-trained model's errors are mostly a single global scale, the baseline will match us. If its errors vary across the image, or it reconstructs what lies behind a glass instead of the glass, per-pixel anchoring should win. MG-Grasp's own failures on reflective objects hint at the second case but do not prove it. We will run this baseline on identical frames. If it matches our millimeter error on opaque objects and our grasp success on transparent ones, we will report that plane-based scale correction is sufficient and that the foot-point head adds nothing.
Why not just use an RGB-D sensor? On opaque objects a depth sensor will likely beat us, and we report it as a reference point, not a target. The case for RGB rests on photometrically hard objects, where the file reports Contact-GraspNet at 6.7% on a cocktail glass. The PhoCaL/HAMMER experiments test whether our method holds up there.
Why not just stay planar and add tilt heuristics? Equivariant Transporter Network and GEM are hard to beat on tasks where a top-down grasp is enough. MetaGrasp and Robot-R1 describe tasks where the approach from above is blocked or orientation must match the object. We include a subset of GraspNet-1Billion scenes where every top-down grasp collides, and run a planar head plus fixed tilt offsets on them. If that heuristic closes the gap, the 6-DoF machinery is unnecessary for these scenes.
Why not just regress the grasp pose or depth directly? PoseNet-scale errors, and MonoGraspNet's own admission that pixel errors become metric errors, argue against it. The depth-regression naive baseline above tests this directly on our features.
Proposed Experiments
On GraspNet-1Billion we report AP on the seen, similar and novel splits for both cameras. We also report, on opaque pixels where sensor depth is valid, the median 3D error in millimeters of lifted points. The lifted-point error is the decisive intermediate check: if foot-point lifting is not more accurate than depth regression and plane-scaled reconstruction there, grasp gains elsewhere would be unexplained.
A sweep over camera elevation, using GraspNet-1Billion's many viewpoints per scene, tests the predicted conditioning effect: error should rise sharply as views approach top-down. On PhoCaL and HAMMER we report grasp success on transparent and reflective objects, split into familiar and novel, against MonoGraspNet's reported numbers and its retrained version.
A real-robot study clears clutter containing glassware and cutlery from a single oblique RGB view. It reports success and latency, and is compared to MG-Grasp's 87.5% protocol with five views. We expect latency well under one second because lifting is closed-form, and we will report the measured value.
The project fails cleanly in any of three cases: the plane-scaled single-view reconstruction matches us, the frozen encoder needs fine-tuning to predict foot points accurately, or errors on stacked objects whose feet are occluded dominate the totals. Each of these outcomes would itself be informative for RGB-only grasping.
References
- MonoGraspNet: 6-DoF Grasping with a Single RGB Image
- MG-Grasp: Metric-Scale Geometric 6-DoF Grasping Framework with Sparse RGB Observations
- Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images
- Scene Coordinate and Correspondence Learning for Image-Based Localization
- PoseNet: A Convolutional Network for Real-Time 6-DOF Camera Relocalization
- 6-DoF object pose from semantic keypoints
- EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation
- The Unsurprising Effectiveness of Pre-Trained Vision Models for Control
- Real-World Robot Learning with Masked Visual Pre-training
- Generalizable Coarse-to-Fine Robot Manipulation via Language-Aligned 3D Keypoints
- Equivariant Transporter Network
- Equivariant Open-vocabulary Pick and Place via Language Kernels and Patch-level Semantic Maps