MMGSD: Multi-Modal Gaussian Shape Descriptors for Correspondence Matching in 1D and 2D Deformable Objects

Aditya Ganapathi, Priya Sundaresan, Brijen Thananjeyan, Ashwin Balakrishna, Daniel Seita, Ryan Hoque, Joseph E. Gonzalez, Ken GoldbergarXiv 2020 · arXiv

Introduction

Robotic manipulation of deformable objects, such as folding laundry or untying knots, presents a significant challenge due to their infinite-dimensional state space and frequent self-occlusions. A major hurdle in this domain is symmetry. When a robot is tasked with identifying a specific feature, such as a corner of a towel, standard vision systems often struggle because multiple points appear identical.

Traditional "dense descriptors" attempt to map a pixel in one image to exactly one pixel in another. Under symmetric conditions, these models often fail by "averaging" locations—predicting a point in the middle of the cloth where no corner exists—or by exhibiting high instability. Furthermore, most existing methods lack a built-in measure of uncertainty, which is critical for safe robotic planning.

Methods: From Single-Point to Distributional Matching

To address these issues, researchers at UC Berkeley developed Multi-Modal Gaussian Shape Descriptors (MMGSD). This approach shifts from unimodal (single-point) matching to a distributional framework.

MMGSD generates probability distributions (heatmaps) to identify multiple symmetric points simultaneously.

Instead of forcing a neural network to map a source pixel to a single target vector, MMGSD learns to predict a probability distribution over the entire image. The model is trained using a cross-entropy loss against a target distribution: an isotropic Gaussian Mixture Model (GMM) centered at all ground-truth symmetric points. This allows the model to "place mass" on all valid matches simultaneously.

A key benefit of this approach is uncertainty estimation. By calculating the entropy of the predicted distribution, the robot can determine its confidence level. High entropy indicates confusion (e.g., the middle of a tangled rope), while low entropy signifies a distinct, identifiable feature (e.g., the end of a rope).

Datasets and Training

The model was trained using a high-fidelity synthetic dataset created in Blender 2.8, consisting of 3,500 training and 500 test images for both square cloth (2D) and braided nylon rope (1D). Synthetic data is essential for this task because it provides perfect ground-truth labels for every "molecule" of the object, which would be impossible to obtain manually in the real world. Domain randomization, including varied textures and lighting, was employed to enhance the model's ability to generalize to real-world scenarios.

Key Results and Insights

MMGSD demonstrates a significant performance leap over standard contrastive learning methods. It achieved a Root-Mean-Square Error (RMSE) of 32.4 for cloth and 31.3 for rope, representing an average 47.7% improvement over the baseline.

Comparison of the contrastive loss baseline (A), ground truth distributions (B), and MMGSD predictions (C). MMGSD successfully captures multiple modes of symmetry.

Qualitatively, MMGSD successfully produces distinct "hotspots" for all symmetric points—such as the four corners of a cloth or the two ends of a rope—whereas previous methods often failed to identify all modes or produced noisy, unstable results. The model also maintains spatial continuity; the generated heatmaps move smoothly as the source pixel is shifted, a property the authors call "distributional continuity."

Conclusion and Future Work

MMGSD provides a robust framework for handling symmetry and uncertainty in deformable object manipulation. By embracing symmetry as a feature rather than a limitation, it allows robots to recognize that any symmetric point (like any corner of a napkin) is a valid target for a task.

While the model represents a major step forward, it still faces challenges in high-occlusion scenarios, such as complex knots. Future work will focus on extending the framework to 3D representations using geodesic distances and testing the system on more complex multi-modal objects like clothing with sleeves and pockets.

Summary updated

Abstract

We explore learning pixelwise correspondences between images of deformable objects in different configurations. Traditional correspondence matching approaches such as SIFT, SURF, and ORB can fail to provide sufficient contextual information for fine-grained manipulation. We propose Multi-Modal Gaussian Shape Descriptor (MMGSD), a new visual representation of deformable objects which extends ideas from dense object descriptors to predict all symmetric correspondences between different object configurations. MMGSD is learned in a self-supervised manner from synthetic data and produces correspondence heatmaps with measurable uncertainty. In simulation, experiments suggest that MMGSD can achieve an RMSE of 32.4 and 31.3 for square cloth and braided synthetic nylon rope respectively. The results demonstrate an average of 47.7% improvement over a provided baseline based on contrastive learning, symmetric pixel-wise contrastive loss (SPCL), as opposed to MMGSD which enforces distributional continuity.
BibTeX
@misc{ganapathi2020mmgsd,
  author = {Aditya Ganapathi and Priya Sundaresan and Brijen Thananjeyan and Ashwin Balakrishna and Daniel Seita and Ryan Hoque and Joseph E. Gonzalez and Ken Goldberg},
  title = {{MMGSD: Multi-Modal Gaussian Shape Descriptors for Correspondence Matching in 1D and 2D Deformable Objects}},
  year = {2020},
  eprint = {2010.04339},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.04339},
}
Figure