Abstract
While Physics-Based Simulation (PBS) can accurately drape a 3D garment on a 3D body, it remains too costly for real-time applications, such as virtual try-on. By contrast, inference in a deep network, requiring a single forward pass, is much faster. Taking advantage of this, we propose a novel architecture to fit a 3D garment template to a 3D body. Specifically, we build upon the recent progress in 3D point cloud processing with deep networks to extract garment features at varying levels of detail, including point-wise, patch-wise and global features. We fuse these features with those extracted in parallel from the 3D body, so as to model the cloth-body interactions. The resulting two-stream architecture, which we call as GarNet, is trained using a loss function inspired by physics-based modeling, and delivers visually plausible garment shapes whose 3D points are, on average, less than 1 cm away from those of a PBS method, while running 100 times faster. Moreover, the proposed method can model various garment types with different cutting patterns when parameters of those patterns are given as input to the network.
Introduction
Draping a 3D garment onto a human body is a computationally demanding task. Traditional Physics-Based Simulation (PBS) calculates complex interactions between cloth fibers and body surfaces, yielding high realism but at a significant cost—often taking seconds or minutes per frame. This latency makes PBS unsuitable for real-time applications like virtual fitting rooms, interactive gaming, or mobile AR.
Previous deep learning approaches often struggled with maintaining fine details or required extensive post-processing to resolve "interpenetration," where cloth visually clips through the skin. GarNet addresses these challenges by providing a fast, inference-only model that handles irregular 3D data while respecting physical constraints.
The GarNet Architecture
GarNet utilizes a two-stream architecture to jointly process the geometry of both the body and the garment.
- Body Stream: Extracts features from the 3D body point cloud using a PointNet-inspired architecture.
- Garment Stream: Processes an initial rough drape (created via Dual Quaternion Skinning) using mesh convolutions. These convolutions analyze a vertex's "1-ring" neighbors to capture local patch-wise geometry, which is critical for modeling realistic wrinkles.
The authors distinguish between two variants: GarNet-Global, which uses a single global feature vector for the body, and GarNet-Local, which employs a K-Nearest Neighbor (K-NN) pooling layer. In the local version, for every garment vertex, the network identifies the nearest body vertices and pools their features, allowing the model to explicitly account for local cloth-body proximity.
To ensure physical plausibility without an iterative solver, GarNet is trained with a physics-inspired loss function. This includes a data term for accuracy, an interpenetration term to prevent skin clipping, and bending/normal terms to maintain realistic cloth curvature and smoothness.
Experimental Results
GarNet demonstrates a significant leap in performance over traditional methods. It achieves inference in under 70ms, representing a 100x to 140x speedup compared to PBS (which typically exceeds 10,000ms). Despite this speed, accuracy remains high: predicted garment shapes are, on average, less than 1 cm away from PBS ground truth.
On public benchmarks for varying sewing patterns, GarNet achieved a 0.89% error rate, a substantial improvement over the previous state-of-the-art error of 3.01%. Qualitatively, the GarNet-Local variant produces drapes nearly indistinguishable from full simulations, effectively avoiding artifacts like spurious wrinkles or clipping.
Conclusion
GarNet provides a robust solution for real-time 3D cloth draping by combining mesh convolutions with local feature pooling. While it occasionally smooths out extremely fine-grained wrinkles due to its regression-based nature, it offers a compelling balance of speed and accuracy. Future work may incorporate Conditional Generative Adversarial Networks (GANs) to recover high-frequency details and explore mesoscopic-scale augmentation for enhanced surface realism.
Summary updated