MG-Grasp: Metric-Scale Geometric 6-DoF Grasping Framework with Sparse RGB Observations

Kangxu Wang, Siang Chen, Chenxing Jiang, Shaojie Shen, Yixiang Dai, Guijin Wang

MG-Grasp: Bringing Metric Precision to Depth-Free Robotic Grasping

TL;DR

MG-Grasp is a novel framework that enables robots to perform high-quality 6-DoF grasping using only a few sparse RGB images, eliminating the need for expensive depth sensors. By combining 3D foundation models with a specialized metric scale recovery and multi-view refinement process, it achieves state-of-the-art performance for RGB-only grasping and significantly narrows the gap with traditional RGB-D systems.

What Problem Does This Paper Address?

Most high-performing robotic grasping systems rely on RGB-D cameras to provide explicit 3D geometry. However, depth sensors add cost, complexity, and often fail on challenging surfaces like reflective or transparent objects.

While "RGB-only" grasping is an active area of research, it faces two major hurdles:

  1. Metric Scale: Neural networks are great at predicting the shape of a scene from images, but they struggle to determine the absolute size (e.g., is that a toy car or a real car?). For a robot to actually grab something, it needs to know distances in centimeters, not just relative proportions.
  2. Cross-View Consistency: When looking at an object from multiple angles, independent predictions often don't "line up" perfectly in 3D space, leading to "ghosting" or misaligned surfaces that confuse grasp planners.

MG-Grasp tackles these issues to generate physically reliable 6-DoF (position and orientation) grasps from sparse, multi-view RGB observations.

Methods

The MG-Grasp pipeline consists of three main stages:

  1. Metric-Scale Depth Map Generator: The system starts with a 3D foundation model (MASt3R) to get initial "up-to-scale" depth maps. To ground these in reality, the authors introduce a Triangulation-based Scale Recovery scheme. Since the camera's relative poses are known, they triangulate matched pixels across views to calculate a global metric scale factor, converting relative depth into real-world units.
  2. Multi-View Guided Refinement: To fix misalignments, the framework uses a two-stage optimization.
  3. Stage I (3D Consistency): Minimizes the distance between matched 3D points across views using a robust Huber loss.
  4. Stage II (2D Reprojection): Further "tightens" the geometry by ensuring that 3D points, when projected back onto the 2D images, land exactly where they should.
  5. Grasp Generation: The refined, fused point cloud is fed into a Local Grasp (LoG) module. This module is "grasp-centric," meaning it samples region centers and predicts gripper poses (position, orientation, and width) based on local geometry, which is more robust to minor global reconstruction errors.

Datasets

  • GraspNet-1Billion: A massive, standard benchmark for robotic grasping. The authors evaluated on the "Seen," "Similar," and "Novel" object splits using both RealSense and Kinect camera data.
  • Real-World Setup: The authors validated the system using a UR5e robot arm equipped with a Robotiq 2F-85 gripper and a standard RGB camera, testing on 25 different daily-use objects.

Key Insights and Ideas

  • Decoupling is Key: Unlike previous methods that try to optimize scale and geometry simultaneously, MG-Grasp decouples them. Recovering the metric scale first provides a much more stable initialization for the subsequent refinement.
  • The "Sweet Spot" for Views: The researchers found that grasping performance peaks at around 5–6 views. Using only 2 views leaves the geometry incomplete, while using more than 6 introduces redundant data and increases latency without significant gains in accuracy.
  • Confidence-Weighted Optimization: By leveraging the confidence scores produced by the foundation model, the refinement process can "ignore" unreliable pixels (like those on edges or shadows), leading to much cleaner 3D reconstructions.

Key Results

  • State-of-the-Art: On the GraspNet-1Billion benchmark, MG-Grasp outperformed the previous best RGB-only method (VG-Grasp) by +12.18/+14.33 AP (Average Precision) across different camera splits.
  • Closing the Gap: It significantly narrowed the performance gap between RGB-only and high-end RGB-D methods (like FlexLoG), despite having no access to actual depth hardware.
  • Efficiency: The system runs with an end-to-end latency of approximately 2.86 seconds (with 5 views), making it fast enough for practical tabletop applications.
  • Real-World Success: In physical robot trials, MG-Grasp achieved an 87.5% success rate in clearing cluttered scenes of daily objects.

Limitations and Future Work

The authors note that the system still struggles with highly reflective or near-spherical objects, as these make it difficult to find the reliable "dense correspondences" needed for accurate 3D reconstruction.

Future work will focus on:

  • Speed: Further accelerating the pipeline through more efficient multi-view selection and GPU-friendly fusion.
  • Closed-Loop Control: Moving from "static" grasping (look then move) to "closed-loop" grasping, where the robot continuously updates its geometry as it approaches the object.

Summary updated

Abstract

Single-view RGB-D grasp detection remains a common choice in 6-DoF robotic grasping systems, which typically requires a depth sensor. While RGB-only 6-DoF grasp methods has been studied recently, their inaccurate geometric representation is not directly suitable for physically reliable robotic manipulation, thereby hindering reliable grasp generation. To address these limitations, we propose MG-Grasp, a novel depth-free 6-DoF grasping framework that achieves high-quality object grasping. Leveraging two-view 3D foundation model with camera intrinsic/extrinsic, our method reconstructs metric-scale and multi-view consistent dense point clouds from sparse RGB images and generates stable 6-DoF grasp. Experiments on GraspNet-1Billion dataset and real world demonstrate that MG-Grasp achieves state-of-the-art (SOTA) grasp performance among RGB-based 6-DoF grasping methods.
BibTeX
@misc{wang2026mg,
  author = {Kangxu Wang and Siang Chen and Chenxing Jiang and Shaojie Shen and Yixiang Dai and Guijin Wang},
  title = {{MG-Grasp: Metric-Scale Geometric 6-DoF Grasping Framework with Sparse RGB Observations}},
  year = {2026},
  eprint = {2603.16270},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.16270},
}
Figure