3D Virtual Garment Modeling from RGB Images

Yi Xu, Shanglin Yang, Wei Sun, Li Tan, Kefeng Li, Hui Zhou

Abstract

We present a novel approach that constructs 3D virtual garment models from photos. Unlike previous methods that require photos of a garment on a human model or a mannequin, our approach can work with various states of the garment: on a model, on a mannequin, or on a flat surface. To construct a complete 3D virtual model, our approach only requires two images as input, one front view and one back view. We first apply a multi-task learning network called JFNet that jointly predicts fashion landmarks and parses a garment image into semantic parts. The predicted landmarks are used for estimating sizing information of the garment. Then, a template garment mesh is deformed based on the sizing information to generate the final 3D model. The semantic parts are utilized for extracting color textures from input images. The results of our approach can be used in various Virtual Reality and Mixed Reality applications.

Introduction

Creating 3D digital fashion assets is traditionally a significant bottleneck in the industry. High-end methods often require specialized multi-camera rigs or expert knowledge of CAD software and 2D sewing patterns. While recent AI-driven methods have emerged, most assume the garment is being worn by a person or a mannequin in a specific, controlled pose.

This research addresses the need for a system that works with "readily available" photos—the kind typically found on e-commerce sites. These images vary wildly; clothes might be on a model, a mannequin, or even laid flat on a table. The goal is to turn these common 2D images into 3D models without requiring manual design or specialized hardware.

Overview of the 3D garment modeling pipeline showing input photo sets and resulting 3D textured models for various garment states.

Methodology

The proposed pipeline consists of three primary stages: Garment Image Parsing, 3D Model Construction, and Texture Extraction.

1. Garment Image Parsing (JFNet)

The core of the system is JFNet, a multi-task learning network built on a ResNet-101 backbone. It performs two tasks simultaneously:

  • Landmark Prediction: Identifying key points like collar corners, sleeve ends, and hemlines.
  • Semantic Segmentation: Labeling pixels as "left sleeve," "torso," "collar," etc.

The network uses a refinement stage where the results of both tasks are concatenated, allowing the structural information from segmentation to improve landmark accuracy.

2. 3D Model Construction

The system uses base meshes from the Berkeley Garment Libraries as templates. Instead of building a mesh from scratch, the system employs Free-Form Deformation (FFD). It calculates sizing (e.g., sleeve length, chest width) from the predicted landmarks and deforms the template mesh to match those specific dimensions.

3. Texture Extraction

To map the garment's appearance onto the 3D model, the system maps the 2D semantic parts onto a 2D reference mesh (UV map). They use Moving Least Squares (MLS) with similarity deformation to warp the textures. This handles the non-rigid nature of cloth better than simple projection, ensuring that textures from folded or posed sleeves in a photo fit correctly onto the 3D mesh.

The proposed pipeline for 3D garment reconstruction, utilizing JFNet for joint landmark and part prediction followed by mesh deformation and texture mapping.

Key Results

The authors compared JFNet against state-of-the-art (SOTA) individual task models like Convolutional Pose Machines (CPM) and DeepLabV3+:

  • Landmark Accuracy: On tops, JFNet achieved a Normalized Error (NE) of 0.031, significantly outperforming CPM's 0.075.
  • Segmentation Quality: JFNet achieved a mean Intersection over Union (mIOU) of 0.725 for tops and 0.968 for pants, showing a slight edge over DeepLabV3+.
  • Qualitative Success: The system successfully generated visually plausible 3D models from various input types (flat lay, mannequin, and human model), demonstrating its flexibility for e-commerce applications.

Conclusion and Future Work

By leveraging multi-task synergy and template-based deformation, this approach provides a robust solution for generating 3D garment models from standard RGB images. While currently limited by template variety and symmetry assumptions, the system demonstrates significant potential for VR and e-commerce. Future work may involve using sewing patterns as an intermediate step to create even more accurate 3D shapes for virtual try-on applications.

Summary updated

Abstract

We present a novel approach that constructs 3D virtual garment models from photos. Unlike previous methods that require photos of a garment on a human model or a mannequin, our approach can work with various states of the garment: on a model, on a mannequin, or on a flat surface. To construct a complete 3D virtual model, our approach only requires two images as input, one front view and one back view. We first apply a multi-task learning network called JFNet that jointly predicts fashion landmarks and parses a garment image into semantic parts. The predicted landmarks are used for estimating sizing information of the garment. Then, a template garment mesh is deformed based on the sizing information to generate the final 3D model. The semantic parts are utilized for extracting color textures from input images. The results of our approach can be used in various Virtual Reality and Mixed Reality applications.
BibTeX
@misc{xu2019d,
  author = {Yi Xu and Shanglin Yang and Wei Sun and Li Tan and Kefeng Li and Hui Zhou},
  title = {{3D Virtual Garment Modeling from RGB Images}},
  year = {2019},
  eprint = {1908.00114},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.00114},
}
Figure