UniGarmentManip: A Unified Framework for Category-Level Garment Manipulation via Dense Visual Correspondence

Ruihai Wu, Haoran Lu, Yiyan Wang, Yubo Wang, Hao DongCVPR 2024 · DOI · OpenReview

Introduction

Manipulating garments—unfolding a messy pile of laundry, folding a shirt, or hanging up a dress—remains a significant challenge in home robotics. Unlike rigid objects, clothing possesses nearly infinite degrees of freedom, frequently self-occludes, and undergoes drastic shape changes during handling. Most existing robotic solutions are task-specific, designed for a single action (like folding) on a specific garment geometry.

Researchers from Peking University and BIT have introduced UniGarmentManip, a unified framework that shifts away from task-specific policies. By teaching robots to understand the underlying topological structure shared by all garments within a category, a single model can handle multiple tasks and generalize to unseen clothing styles with minimal human guidance.

UniGarmentManip framework overview: Transferring manipulation skills across garments via dense correspondence.

Methodology

The core innovation of UniGarmentManip is the use of self-supervised dense visual correspondence to map the "identity" of points across different garment shapes and deformations. The framework operates in three primary stages:

1. Learning Topological Correspondence

The learning process is decoupled into two distinct components:

  • Cross-Deformation: Using "self-play" in simulation, the model learns to identify that a point on a flat shirt remains the same point when crumpled. This is trained using an InfoNCE loss, a contrastive learning technique that aligns matching points in feature space while separating non-matching ones.
  • Cross-Object: To bridge different garments (e.g., from a T-shirt to a jacket), the system employs a Skeleton Merger. This extracts a sparse graph of keypoints representing the garment's topology. By interpolating between these points, the model creates a dense map that aligns different garments within the same category.

2. Coarse-to-Fine (C2F) Refinement

Initial correspondences can be imprecise, particularly near edges. The authors introduce an online refinement stage that identifies "failure points"—regions where the model is confident but geographically inaccurate in the canonical (flat) state—and applies additional training weight to those areas.

3. Few-Shot Functional Adaptation

Because a topological point (like a sleeve) is not always the optimal functional point for a specific task (like where to grab for folding), the model undergoes few-shot adaptation. With only 1–5 human demonstrations, the model fine-tunes its dense representations to prioritize the specific points required for tasks like hanging or folding.

The UniGarmentManip pipeline: From self-supervised correspondence learning to multi-task execution.

Experimental Results

The framework was evaluated using the CLOTH3D dataset and real-world physical tests involving Franka Panda arms. UniGarmentManip was compared against state-of-the-art baselines such as FlingBot and ClothFunnels.

  • Unfolding: The system achieved success rates between 83.6% and 86.9% across categories, outperforming FlingBot, which struggled with diverse garment scales.
  • Folding: In the "Fold-FLING" task (starting from a messy state), UniGarmentManip achieved an ~82% success rate, significantly higher than ClothFunnels (~62%), which typically requires garments to be pre-flattened.
  • Real-World Execution: In physical trials, the robot successfully folded five different real-world tops with an 11/15 success rate, demonstrating robust sim-to-real generalization.

Conclusion

UniGarmentManip demonstrates that leveraging topological consistency is a powerful strategy for deformable object manipulation. By decoupling identity from state and using skeletons as a topological bridge, the framework allows robots to reuse manipulation logic across entire categories of clothing. While challenges remain regarding extreme occlusions and sim-to-real gaps for complex textures, this work represents a significant step toward general-purpose laundry robotics.

Summary updated

Abstract

Garment manipulation (e.g., unfolding, folding and hanging clothes) is essential for future robots to accomplish home-assistant tasks, while highly challenging due to the diversity of garment configurations, geometries and deformations. Although able to manipulate similar shaped garments in a certain task, previous works mostly have to design different policies for different tasks, could not generalize to garments with diverse geometries, and often rely heavily on human-annotated data. In this paper, we leverage the property that, garments in a certain category have similar structures, and then learn the topological dense (point-level) visual correspondence among garments in the category level with different deformations in the self-supervised manner. The topological correspondence can be easily adapted to the functional correspondence to guide the manipulation policies for various downstream tasks, within only one or few-shot demonstrations. Experiments over garments in 3 different categories on 3 representative tasks in diverse scenarios, using one or two arms, taking one or more steps, inputting flat or messy garments, demonstrate the effectiveness of our proposed method. Project page: https://warshallrho.github.io/unigarmentmanip.
BibTeX
@inproceedings{wu2024unigarmentmanip,
  author = {Ruihai Wu and Haoran Lu and Yiyan Wang and Yubo Wang and Hao Dong},
  title = {{UniGarmentManip: A Unified Framework for Category-Level Garment Manipulation via Dense Visual Correspondence}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01546},
}
Figure