MG-Grasp: Bringing Metric Precision to Depth-Free Robotic Grasping
TL;DR
MG-Grasp is a novel framework that enables robots to perform high-quality 6-DoF grasping using only a few sparse RGB images, eliminating the need for expensive depth sensors. By combining 3D foundation models with a specialized metric scale recovery and multi-view refinement process, it achieves state-of-the-art performance for RGB-only grasping and significantly narrows the gap with traditional RGB-D systems.
What Problem Does This Paper Address?
Most high-performing robotic grasping systems rely on RGB-D cameras to provide explicit 3D geometry. However, depth sensors add cost, complexity, and often fail on challenging surfaces like reflective or transparent objects.
While "RGB-only" grasping is an active area of research, it faces two major hurdles:
- Metric Scale: Neural networks are great at predicting the shape of a scene from images, but they struggle to determine the absolute size (e.g., is that a toy car or a real car?). For a robot to actually grab something, it needs to know distances in centimeters, not just relative proportions.
- Cross-View Consistency: When looking at an object from multiple angles, independent predictions often don't "line up" perfectly in 3D space, leading to "ghosting" or misaligned surfaces that confuse grasp planners.
MG-Grasp tackles these issues to generate physically reliable 6-DoF (position and orientation) grasps from sparse, multi-view RGB observations.
Methods
The MG-Grasp pipeline consists of three main stages:
- Metric-Scale Depth Map Generator: The system starts with a 3D foundation model (MASt3R) to get initial "up-to-scale" depth maps. To ground these in reality, the authors introduce a Triangulation-based Scale Recovery scheme. Since the camera's relative poses are known, they triangulate matched pixels across views to calculate a global metric scale factor, converting relative depth into real-world units.
- Multi-View Guided Refinement: To fix misalignments, the framework uses a two-stage optimization.
- Stage I (3D Consistency): Minimizes the distance between matched 3D points across views using a robust Huber loss.
- Stage II (2D Reprojection): Further "tightens" the geometry by ensuring that 3D points, when projected back onto the 2D images, land exactly where they should.
- Grasp Generation: The refined, fused point cloud is fed into a Local Grasp (LoG) module. This module is "grasp-centric," meaning it samples region centers and predicts gripper poses (position, orientation, and width) based on local geometry, which is more robust to minor global reconstruction errors.
Datasets
- GraspNet-1Billion: A massive, standard benchmark for robotic grasping. The authors evaluated on the "Seen," "Similar," and "Novel" object splits using both RealSense and Kinect camera data.
- Real-World Setup: The authors validated the system using a UR5e robot arm equipped with a Robotiq 2F-85 gripper and a standard RGB camera, testing on 25 different daily-use objects.
Key Insights and Ideas
- Decoupling is Key: Unlike previous methods that try to optimize scale and geometry simultaneously, MG-Grasp decouples them. Recovering the metric scale first provides a much more stable initialization for the subsequent refinement.
- The "Sweet Spot" for Views: The researchers found that grasping performance peaks at around 5–6 views. Using only 2 views leaves the geometry incomplete, while using more than 6 introduces redundant data and increases latency without significant gains in accuracy.
- Confidence-Weighted Optimization: By leveraging the confidence scores produced by the foundation model, the refinement process can "ignore" unreliable pixels (like those on edges or shadows), leading to much cleaner 3D reconstructions.
Key Results
- State-of-the-Art: On the GraspNet-1Billion benchmark, MG-Grasp outperformed the previous best RGB-only method (VG-Grasp) by +12.18/+14.33 AP (Average Precision) across different camera splits.
- Closing the Gap: It significantly narrowed the performance gap between RGB-only and high-end RGB-D methods (like FlexLoG), despite having no access to actual depth hardware.
- Efficiency: The system runs with an end-to-end latency of approximately 2.86 seconds (with 5 views), making it fast enough for practical tabletop applications.
- Real-World Success: In physical robot trials, MG-Grasp achieved an 87.5% success rate in clearing cluttered scenes of daily objects.
Limitations and Future Work
The authors note that the system still struggles with highly reflective or near-spherical objects, as these make it difficult to find the reliable "dense correspondences" needed for accurate 3D reconstruction.
Future work will focus on:
- Speed: Further accelerating the pipeline through more efficient multi-view selection and GPU-friendly fusion.
- Closed-Loop Control: Moving from "static" grasping (look then move) to "closed-loop" grasping, where the robot continuously updates its geometry as it approaches the object.
Summary updated