TL;DR
This paper introduces Deep Fashion3D, the largest dataset to date for 3D garment reconstruction from single images, addressing the critical lack of data in this area. They also propose a novel baseline approach that combines mesh and implicit representations to achieve state-of-the-art results in reconstructing realistic 3D garments from a single image.
What Problem Does This Paper Address?
Reconstructing realistic 3D human bodies from images has seen significant progress, largely due to the availability of large datasets like SMPL. However, reconstructing clothed humans is considerably more challenging. This is primarily due to a scarcity of large-scale 3D garment datasets, the complex deformations clothing undergoes, and the diverse topologies of garments. Existing methods often struggle with realism, fine details, and generalizing to unseen garment types. This paper aims to bridge this gap by introducing a comprehensive dataset and a robust baseline for single-view 3D garment reconstruction.
Methods
The authors propose a multi-stage pipeline for single-view 3D garment reconstruction:
- Adaptable Template: Instead of using category-specific templates, they introduce a single, adaptable template mesh that can scale to various garment topologies. This template is derived from the SMPL model, with specific regions activated based on the predicted clothing category.
- Cloth Classification: A VGGNet-based classifier is trained to predict the garment category, which in turn activates the appropriate regions of the adaptable template.
- Pose Estimation: The body pose is estimated using an SMPL model, which helps to align the template mesh to the input image.
- Feature Line-Guided Mesh Generation: They introduce "3D feature lines" (e.g., necklines, hemlines) as a novel annotation. These lines are predicted using a graph convolutional network (GCN) and then used as "handles" to deform the template mesh, guiding the reconstruction.
- Surface Refinement with Implicit Reconstruction: To capture fine geometric details, they employ an implicit surface reconstruction method (OccNet). The output from OccNet is then adaptively registered to the mesh generated from feature lines to transfer high-fidelity details while preserving the overall structure.
Datasets
- Deep Fashion3D: The paper introduces this as a new, large-scale dataset for 3D garment reconstruction. It contains over 2000 3D garment models reconstructed from real garments, covering 10 categories and 563 instances. Annotations include multi-view real images, 3D body pose, and the novel 3D feature lines. Garments are also randomly posed to capture diverse deformations.
- Existing Datasets (for comparison/training): The paper references and compares against datasets like BUFF, MGN, Wang et al., and synthetic datasets generated through physical simulation.
Key Insights and Ideas
- The Power of Feature Lines: Introducing explicit "3D feature lines" as annotations provides strong, localized priors for garment reconstruction, significantly aiding in capturing key contours and shapes.
- Unified Adaptable Template: Moving away from category-specific templates to a single, adaptable template that can handle diverse topologies is crucial for training a single, expressive network. This improves generalization and avoids overfitting issues associated with smaller, category-specific datasets.
- Hybrid Mesh-Implicit Approach: Combining the strengths of mesh-based representations (handling open boundaries) and implicit representations (capturing fine details) is key to achieving realistic and accurate 3D garment reconstruction.
- Data Scarcity is the Bottleneck: The paper strongly emphasizes that the lack of large, well-annotated 3D garment datasets is the primary impediment to progress in this field.
Key Results
- State-of-the-Art Performance: The proposed baseline approach, trained on Deep Fashion3D, achieves state-of-the-art results on single-view garment reconstruction benchmarks.
- Quantitative Improvements: On their testing dataset, their method achieves the lowest Chamfer Distance (CD) of 0.679 (×10⁻³) and Earth Mover's Distance (EMD) of 2.942 (×10²), outperforming methods like 3D-R2N2, MVD, Pixel2Mesh, AtlasNet, TMN, and OccNet.
- Qualitative Success: The method demonstrates impressive qualitative results, accurately capturing garment topologies and geometric details from diverse input images and poses, including "in-the-wild" images. Ablation studies show the effectiveness of each component of their pipeline.
Limitations and Future Work
- Front-View Focus: The current approach primarily focuses on front-view reconstruction. The authors note that their approach can be extended to side or back views by providing corresponding training images.
- GCN for Geometric Details: The paper observes that GCNs, while useful, struggle to synthesize high-frequency geometric details while maintaining surface smoothness. They suggest this is an area for further study.
- In-the-Wild Generalization: While the results on in-the-wild images are promising, the paper acknowledges that single-view garment reconstruction remains an extremely challenging problem with ample room for improvement.
- Future Research: The authors hope that Deep Fashion3D and their baseline approach will inspire future research in this field.
Summary updated