MMRDN: Consistent Representation for Multi-View Manipulation Relationship Detection in Object-Stacked Scenes

Han Wang, Jiayuan Zhang, Lipeng Wan, Xingyu Chen, Xuguang Lan, Nanning Zheng

Abstract

Manipulation Relationship Detection (MRD) is critical for robotic tasks involving stacked objects, where determining the correct grasping order prevents stack collapse. Traditional single-view methods often fail due to visual occlusions and perspective-induced errors. We present the Multi-View Manipulation Relationship Network (MMRDN), a framework that fuses 2D RGB and 3D point cloud data from multiple viewpoints. By employing Von-Mises-Fisher distributions for feature alignment and a novel "K Maximum Vertical Neighbors" (KMVN) module for 3D spatial analysis, MMRDN achieves a consistent representation across diverse camera angles. Experimental results on the REGRAD dataset and real-world robotic platforms demonstrate state-of-the-art performance and robust generalization to unseen camera positions.

Introduction

In robotic manipulation, identifying "parent-child" relationships—where one object supports another—is essential for planning safe grasping sequences. If a robot attempts to remove a supporting "parent" object before its "child," the resulting instability can lead to task failure or damage. While deep learning has advanced MRD using single-view RGB images, these approaches are inherently limited by visual dislocation and occlusions.

Using multiple cameras can mitigate these issues, but it introduces a significant "domain shift" challenge: the same physical relationship appears drastically different across various camera perspectives. MMRDN addresses this by learning a view-invariant, consistent representation of object relationships, ensuring that the underlying physical structure is correctly interpreted regardless of the camera's vantage point.

Overview of the MMRDN framework showing consistent representation across seen and unseen views.

Methodology

The MMRDN architecture integrates 2D and 3D data through three primary components designed to ensure cross-view consistency and geometric accuracy.

Detailed architecture of the Multi-View Manipulation Relationship Network (MMRDN).

1. Image-level Alignment Module

To mitigate the domain shift between different camera views, the network incorporates a Gradient Reversal Layer (GRL). During training, the GRL encourages the backbone network to extract features that are discriminative for relationship detection but invariant to the specific camera viewpoint. This "view-blind" feature extraction is the first step toward a unified representation.

2. Instance-level Alignment Module

This module aligns object features in a common latent space by fitting them to Von-Mises-Fisher (VMF) distributions. VMF distributions, which operate on a unit sphere, allow the model to focus on the "direction" of relationship embeddings rather than their magnitude, which can be influenced by lighting or perspective. By minimizing the distance between these distributions across views, the model ensures that the parent-child relationship remains consistent in the embedding space.

3. Contact Point Module (CPM)

Traditional 3D methods often rely on object centroids to determine vertical relationships, which can be misleading for irregularly shaped objects. MMRDN introduces the K Maximum Vertical Neighbors (KMVN) module. KMVN identifies the $K$ points in the point cloud with the highest relative vertical angles between two objects, focusing the network's attention on the actual contact area. This geometric approach provides a more reliable indicator of support than simple centroid comparisons.

Experiments and Results

The model was evaluated using the REGRAD dataset, a large-scale synthetic benchmark featuring nine camera views of object-stacked scenes. Training was conducted on four views, while evaluation spanned both "seen" and "unseen" (novel) camera angles.

Performance and Generalization

MMRDN significantly outperformed existing state-of-the-art models such as VMRN and GGNN-VMRN. Key results include:

  • Scene Accuracy (SA): An improvement of 5.6% over previous methods.
  • Relationship Recall: On the REGRAD dataset, MMRDN achieved a parent relationship recall of 43.48% on seen views and 46.33% on unseen views. In contrast, the baseline VMRN achieved only ~16% and ~4%, respectively.
  • Sim-to-Real Transfer: Despite being trained exclusively on synthetic data, the model maintained high precision when tested on real-world robotic hardware, demonstrating its robustness to physical noise and lighting variations.

Discussion and Conclusion

MMRDN demonstrates that combining spherical embeddings with targeted 3D geometric analysis can overcome the limitations of single-view manipulation relationship detection. The KMVN module effectively bypasses the pitfalls of centroid-based reasoning, while VMF alignment ensures robustness across camera placements.

However, challenges remain. The current inference time is approximately 0.865 seconds per scene, with point cloud processing (KMVN) accounting for 0.682 seconds. This latency may limit applications in high-speed reactive grasping. Additionally, while the model excels in moderately complex scenes, its performance decreases in highly cluttered environments with more than ten stacked objects. Future research will focus on optimizing point cloud processing and incorporating advanced graph reasoning to handle extreme clutter.

Summary updated

BibTeX
@inproceedings{wang2023mmrdn,
  author = {Han Wang and Jiayuan Zhang and Lipeng Wan and Xingyu Chen and Xuguang Lan and Nanning Zheng},
  title = {{MMRDN: Consistent Representation for Multi-View Manipulation Relationship Detection in Object-Stacked Scenes}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=Hpo_noQUS3},
}
Figure