MMGNet: Multi-Modal 6-DoF Grasp Synthesis for Cluttered Scenes

Rui Cao, Jia-Hui Pan, Yichuan Li, Yunfan Yu, Chi-Wing Fu, Jiangliu Wang, Yun-Hui LiuarXiv 2026 · arXiv

TL;DR

MMGNet is a new multi-modal framework that significantly boosts robotic grasping performance in cluttered scenes by deeply integrating RGB images with 3D point clouds. By injecting multi-scale image features directly into a sparse 3D backbone and introducing a "rotational graspness" model, it achieves state-of-the-art results on the GraspNet-1Billion benchmark and improves real-world grasp success rates by over 8% on challenging objects.

What Problem Does This Paper Address?

In the world of robotic manipulation, "6-DoF grasp synthesis" is the task of predicting the exact 3D position and orientation a gripper should take to pick up an object. Most current systems rely heavily on point clouds derived from depth sensors. However, real-world depth data is often "messy"—it can be noisy, incomplete (due to occlusions), or over-smoothed, making it hard to distinguish object boundaries or fine details.

While RGB images provide rich cues about texture, boundaries, and appearance that could help, effectively combining 2D images with 3D points is difficult. Previous attempts usually suffered from two bottlenecks:

  1. Shallow Fusion: They often combined image and 3D data only at the very beginning or the very end, missing out on the benefits of multi-scale context.
  2. Coarse Reasoning: They typically picked a grasp location first and only considered the gripper's rotation later, meaning the local features used for prediction weren't actually "aware" of the gripper's orientation.

Methods

MMGNet (Multi-Modal Grasp Synthesis Network) introduces two primary innovations to break these bottlenecks:

1. Hierarchical Image-to-3D Feature Injection

Instead of a simple "late fusion" approach, MMGNet uses a hierarchical injection strategy. It extracts a pyramid of multi-scale features from the RGB image (using ResNet and PSPNet). These features are then "injected" into corresponding stages of a sparse 3D backbone (MinkowskiEngine). This allows visual information to shape the 3D representation throughout the entire encoding process, rather than just acting as an afterthought.

Figure 1: Illustration of MMGNet. Given a single-view RGB-D observation, the image backbone extracts

2. Rotational Graspness and Grasp-Shaded Grouping

Standard models often predict "view graspness" (how good a point is from a certain approach direction). MMGNet proposes Rotational Graspness, which jointly scores the approach direction and the in-plane rotation of the gripper.

Once a rotation is selected, the model performs Grasp-Shaded Grouping. Unlike traditional methods that gather points in a generic cylinder, this method uses a cuboid region aligned specifically with the predicted gripper orientation. This ensures the local features are organized according to the candidate rotation, capturing the specific geometry the gripper will encounter.

Grasp representation and the rotation-conditioned grasp-shaded grouping.

Datasets

The authors evaluated MMGNet using:

  • GraspNet-1Billion: A massive-scale benchmark for general object grasping, featuring 190 scenes and over 1 billion grasp poses. It includes data from two different sensors: Intel RealSense and Microsoft Kinect.
  • Adversarial Grasp Objects: A specific set of "novel" objects with challenging geometries and colors used to test the limits of the system's robustness in real-world experiments.

Key Insights and Ideas

  • Deep Integration is Key: The authors found that hierarchical injection produces much stronger cross-modal correspondence than early or late fusion. This suggests that 3D features "learn" better when they have access to 2D context at every level of abstraction.
  • Rotation Matters Early: By considering gripper rotation before aggregating local features, the model becomes much more sample-efficient. MMGNet outperformed traditional grouping methods even when using significantly fewer local point samples.
  • RGB as a Safety Net: Experiments with "corrupted" data showed that the image features act as a vital backup. When point clouds were degraded by noise or missing regions, the image-informed features compensated for the lost geometric detail.

Key Results

  • State-of-the-Art Performance: MMGNet achieved a mean Average Precision (AP) of 60.34 on the GraspNet-1Billion RealSense test set, surpassing the previous best method (FlexLoG) by 4.32 points.
  • Real-World Success: In physical robot trials with a Franka Panda arm, MMGNet achieved an 81.43% success rate on adversarial objects, an 8.46 percentage point improvement over the popular AnyGrasp baseline.
  • Efficiency: Despite the extra image processing, MMGNet maintains a practical inference time of roughly 71.74 ms per scene, making it suitable for real-time applications.

Limitations and Future Work

While powerful, MMGNet has a few areas for improvement:

  • Calibration Sensitivity: The system relies on accurate pixel-to-point correspondence. If the camera and depth sensor are poorly calibrated, performance will drop.
  • Computational Overhead: Adding a full image backbone (ResNet34) increases the total parameter count and GFLOPs compared to point-only models.
  • Lack of Explicit Fallback: The current model doesn't have a mechanism to "ignore" the RGB data if the image is extremely low-quality or misleading (e.g., in total darkness).

Future research directions include making the fusion "reliability-aware" and incorporating online feedback to help the robot recover from failed grasp attempts.

Examples of failure modes observed during real-world testing, such as slipping or suboptimal selection.

Summary updated

Abstract

Grasping is a foundational capability in robotic manipulation. Scene-level 6-DoF grasp synthesis relies on point clouds for metric geometry, yet real-world point observations are often noisy, incomplete, and over-smoothed. RGB images provide object-region, boundary, and appearance cues, yet two bottlenecks limit their effective use in scene representation and local grasp reasoning. We present MMGNet, a Multi-Modal Grasp Synthesis Network that addresses these bottlenecks through hierarchical image-to-3D feature injection and rotational graspness modeling. Hierarchical injection integrates multi-scale image features throughout 3D representation formation, while the rotational graspness jointly predicts the grasp approach direction and in-plane rotation before organizing local image-informed features through grasp-shaded grouping. Ablation studies show that these two modules provide complementary gains. MMGNet achieves a mean AP of 60.34 on GraspNet-1Billion with RealSense, surpassing all compared methods. In real-world experiments with adversarial grasp objects, MMGNet improves the grasp success rate over AnyGrasp by 8.46 percentage points. \emph{Code, models, and data will be released upon publication.}
BibTeX
@misc{cao2026mmgnet,
  author = {Rui Cao and Jia-Hui Pan and Yichuan Li and Yunfan Yu and Chi-Wing Fu and Jiangliu Wang and Yun-Hui Liu},
  title = {{MMGNet: Multi-Modal 6-DoF Grasp Synthesis for Cluttered Scenes}},
  year = {2026},
  eprint = {2610.34893},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34893},
}
Figure