Abstract
We present a novel approach that constructs 3D virtual garment models from photos. Unlike previous methods that require photos of a garment on a human model or a mannequin, our approach can work with various states of the garment: on a model, on a mannequin, or on a flat surface. To construct a complete 3D virtual model, our approach only requires two images as input, one front view and one back view. We first apply a multi-task learning network called JFNet that jointly predicts fashion landmarks and parses a garment image into semantic parts. The predicted landmarks are used for estimating sizing information of the garment. Then, a template garment mesh is deformed based on the sizing information to generate the final 3D model. The semantic parts are utilized for extracting color textures from input images. The results of our approach can be used in various Virtual Reality and Mixed Reality applications.
Introduction
Creating 3D digital fashion assets is traditionally a significant bottleneck in the industry. High-end methods often require specialized multi-camera rigs or expert knowledge of CAD software and 2D sewing patterns. While recent AI-driven methods have emerged, most assume the garment is being worn by a person or a mannequin in a specific, controlled pose.
This research addresses the need for a system that works with "readily available" photos—the kind typically found on e-commerce sites. These images vary wildly; clothes might be on a model, a mannequin, or even laid flat on a table. The goal is to turn these common 2D images into 3D models without requiring manual design or specialized hardware.
Methodology
The proposed pipeline consists of three primary stages: Garment Image Parsing, 3D Model Construction, and Texture Extraction.
1. Garment Image Parsing (JFNet)
The core of the system is JFNet, a multi-task learning network built on a ResNet-101 backbone. It performs two tasks simultaneously:
- Landmark Prediction: Identifying key points like collar corners, sleeve ends, and hemlines.
- Semantic Segmentation: Labeling pixels as "left sleeve," "torso," "collar," etc.
The network uses a refinement stage where the results of both tasks are concatenated, allowing the structural information from segmentation to improve landmark accuracy.
2. 3D Model Construction
The system uses base meshes from the Berkeley Garment Libraries as templates. Instead of building a mesh from scratch, the system employs Free-Form Deformation (FFD). It calculates sizing (e.g., sleeve length, chest width) from the predicted landmarks and deforms the template mesh to match those specific dimensions.
3. Texture Extraction
To map the garment's appearance onto the 3D model, the system maps the 2D semantic parts onto a 2D reference mesh (UV map). They use Moving Least Squares (MLS) with similarity deformation to warp the textures. This handles the non-rigid nature of cloth better than simple projection, ensuring that textures from folded or posed sleeves in a photo fit correctly onto the 3D mesh.
Key Results
The authors compared JFNet against state-of-the-art (SOTA) individual task models like Convolutional Pose Machines (CPM) and DeepLabV3+:
- Landmark Accuracy: On tops, JFNet achieved a Normalized Error (NE) of 0.031, significantly outperforming CPM's 0.075.
- Segmentation Quality: JFNet achieved a mean Intersection over Union (mIOU) of 0.725 for tops and 0.968 for pants, showing a slight edge over DeepLabV3+.
- Qualitative Success: The system successfully generated visually plausible 3D models from various input types (flat lay, mannequin, and human model), demonstrating its flexibility for e-commerce applications.
Conclusion and Future Work
By leveraging multi-task synergy and template-based deformation, this approach provides a robust solution for generating 3D garment models from standard RGB images. While currently limited by template variety and symmetry assumptions, the system demonstrates significant potential for VR and e-commerce. Future work may involve using sewing patterns as an intermediate step to create even more accurate 3D shapes for virtual try-on applications.
Summary updated