GarmentNets: Category-Level Pose Estimation for Garments via Canonical Space Shape Completion

Cheng Chi, Shuran Song

Abstract

This paper tackles the task of category-level pose estimation for garments. With a near infinite degree of freedom, a garment’s full configuration (i.e., poses) is often described by the per-vertex 3D locations of its entire 3D surface. However, garments are also commonly subject to extreme cases of self-occlusion, especially when folded or crumpled, making it challenging to perceive their full 3D surface. To address these challenges, we propose GarmentNets, where the key idea is to formulate the deformable object pose estimation problem as a shape completion task in the canonical space. This canonical space is defined across garments instances within a category, therefore, specifies the shared category-level pose. By mapping the observed partial surface to the canonical space and completing it in this space, the output representation describes the garment’s full configuration using a complete 3D mesh with the per-vertex canonical coordinate label. To properly handle the thin 3D structure presented on garments, we proposed a novel 3D shape representation using the generalized winding number field. Experiments demonstrate that GarmentNets is able to generalize to unseen garment instances and achieve significantly better performance compared to alternative approaches.

Introduction

Perceiving garments is a significant challenge for robotic systems. Unlike rigid objects, which can be described by six degrees of freedom (position and orientation), garments possess near-infinite degrees of freedom. When picked up or laid in a pile, clothing items crumple and fold, leading to severe self-occlusion where most of the surface area is hidden from view.

Furthermore, standard 3D representations in computer vision—such as occupancy grids or Signed Distance Functions (SDFs)—are optimized for "water-tight" solid objects. Garments, however, are thin surfaces with functional openings like necklines, sleeves, and waistbands. Traditional volumetric methods often erroneously "close" these holes or fail to capture the thin geometry accurately, hindering a robot's ability to identify grasping points or perform complex tasks like folding.

Methods

GarmentNets addresses these challenges by formulating pose estimation as a shape completion task within a shared canonical space. The framework operates in four distinct stages:

  1. Pick and Observe: Instead of analyzing a flat pile, the robot picks up the garment at a random point. Gravity pulls the cloth into a semi-stable hanging state, reducing the complexity of the configuration.
  2. Canonicalization (NOCS Prediction): Using a PointNet++ backbone, the system predicts Normalized Object Coordinates (NOCS) for the visible partial point cloud. This step maps observed points to their corresponding locations on a standard "T-pose" model for that garment category.
  3. Shape Completion with Winding Number Fields (WNF): The predicted coordinates are used to populate a 3D volume with features. A 3D CNN then completes the missing geometry in the canonical space.
  4. Implicit Warp Field: A separate network learns an implicit warp field that maps the completed canonical mesh back into the real-world "task space" (the hanging pose), providing a full 3D reconstruction of the garment's current state.
Overview of the GarmentNets pipeline: from pickup to full configuration estimation.

Generalized Winding Number Fields

A core technical contribution of GarmentNets is the use of the Generalized Winding Number Field (WNF) for shape representation. Traditional Truncated Signed Distance Functions (TSDF) struggle with non-water-tight surfaces, often creating artificial surfaces across openings. In contrast, the WNF provides a smooth gradient that naturally handles thin, open geometries. By analyzing the magnitude of the spatial gradient, the model can distinguish between the actual cloth surface and open holes like sleeve openings.

Comparison of Winding Number Field (WNF) and Truncated Signed Distance Function (TSDF) for representing thin, open garment surfaces.

Key Results

The framework was evaluated using the CLOTH3D dataset, which contains thousands of synthetic garment instances across various categories. Key results include:

  • Pose Estimation Accuracy: GarmentNets achieved correspondence distance errors as low as 4.37cm to 7.34cm, vastly outperforming direct task-space completion baselines which failed with errors exceeding 30cm.
  • Reconstruction Quality: The WNF representation yielded a 26.7% improvement over TSDF and a 32.2% improvement over standard occupancy grids in reconstruction accuracy.
  • Real-World Generalization: Despite being trained solely on synthetic data, the model successfully reconstructed real-world garments from noisy RGB-D scans captured by a mobile device.

Conclusion

GarmentNets demonstrates that mapping deformable objects to a canonical space is a powerful strategy for handling extreme self-occlusion and high degrees of freedom. While the current system is not yet fully end-to-end trainable, it sets a new benchmark for category-level garment perception. Future work will explore self-supervised learning techniques to reduce reliance on high-quality synthetic labels and enable learning directly from real-world interactions.

Summary updated

Abstract

This paper tackles the task of category-level pose estimation for garments. With a near infinite degree of freedom, a garment’s full configuration (i.e., poses) is often described by the per-vertex 3D locations of its entire 3D surface. However, garments are also commonly subject to extreme cases of self-occlusion, especially when folded or crumpled, making it challenging to perceive their full 3D surface. To address these challenges, we propose GarmentNets, where the key idea is to formulate the deformable object pose estimation problem as a shape completion task in the canonical space. This canonical space is defined across garments instances within a category, therefore, specifies the shared category-level pose. By mapping the observed partial surface to the canonical space and completing it in this space, the output representation describes the garment’s full configuration using a complete 3D mesh with the per-vertex canonical coordinate label. To properly handle the thin 3D structure presented on garments, we proposed a novel 3D shape representation using the generalized winding number field. Experiments demonstrate that GarmentNets is able to generalize to unseen garment instances and achieve significantly better performance compared to alternative approaches. Code and data can be found in https://garmentnets.cs.columbia.edu.
BibTeX
@inproceedings{chi2021garmentnets,
  author = {Cheng Chi and Shuran Song},
  title = {{GarmentNets: Category-Level Pose Estimation for Garments via Canonical Space Shape Completion}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=iYfnX55T76p},
}
Figure