Multi-Layered 3D Garments Animation

Yidi Shao, Chen Change Loy, Bo DaiICLR 2023 · OpenReview

TL;DR

This paper introduces LayersNet, a novel Transformer-based method for animating multi-layered 3D garments, which handles complex interactions between garments, human bodies, and environmental forces like wind. They also release D-LAYERS, a large-scale dataset of dynamic multi-layered garment animations, to bridge the gap between simplified simulations and real-world scenarios.

What Problem Does This Paper Address?

Realistic 3D garment animation is a significant challenge in computer graphics and machine learning. Existing methods often simplify the problem by focusing on single-layered garments driven solely by human body movement. This overlooks the complexities of real-world scenarios where individuals wear multiple layers of clothing, and external factors like wind significantly influence how garments move. These simplifications limit the generalizability of current approaches, making them struggle with multi-layered outfits and diverse environmental conditions. The authors aim to address this gap by developing a method and dataset that can handle these more complex, realistic scenarios.

Methods

The core of LayersNet is a Transformer-based simulation system that models garment dynamics through particle-wise interactions. To manage computational complexity, they introduce a two-level structural hierarchy for garments, representing them as patches (collections of vertices) rather than individual vertices. These patches act as the "particles" for interaction learning.

A key innovation is the Rotation Equivalent Transformation (RET). This technique leverages the physics properties of rotation invariance and additivity. It transforms high-dimensional features into a local canonical space defined by 3D directions (like surface normals), allowing the model to learn interaction semantics more effectively, regardless of the orientation of the garment or external forces. This transformed information is then aggregated.

The system uses an encoder-decoder architecture. The encoder embeds input features (garment attributes, human pose, wind) into a hidden space, and the decoder reconstructs the garment's future state. The training involves a combination of Mean Squared Error (MSE) loss for vertex positions, a loss for garment vertex normals to ensure smoothness, and a collision loss to minimize interpenetration between garments and with the human body.

Datasets

The paper introduces D-LAYERS (Dynamic multi-lAYerEd gaRmentS), a new, large-scale dataset for 3D garment animation.

  • Content: It features 4,900 combinations of multi-layered garments (inner and outer layers with varying attributes like stiffness and friction) driven by both human body movements and randomly sampled wind.
  • Scale: It comprises 700,000 frames in total, with sequences up to 600 frames long.
  • Difference from existing datasets: Unlike previous datasets that often focus on single-layered garments or static poses, D-LAYERS emphasizes dynamic, multi-layered scenarios with external forces, making it more representative of real-world applications.

The dataset was generated using a simulation engine and Blender, with garment meshes scaled up to preserve high-frequency details.

Key Insights and Ideas

  • Patch-based particles simplify complexity: Treating garment patches as the fundamental interacting units significantly reduces computational overhead compared to vertex-level simulations, especially for high-fidelity meshes.
  • Rotation Equivalence is crucial for generalization: By explicitly handling rotation invariance, the model can learn interaction semantics more robustly, leading to better generalization across different garment orientations and external force directions. This is a significant step beyond methods that rely on fixed coordinate frames.
  • Unified interaction modeling: Representing garments, human bodies, and wind as particles allows for a unified approach to modeling their interactions, making the system agnostic to garment topology and the number of layers.
  • D-LAYERS fills a critical gap: The dataset's focus on multi-layered garments and wind provides a much-needed benchmark for evaluating and advancing garment animation techniques in more realistic settings.

Key Results

  • Quantitative Performance: LayersNet consistently outperforms baseline methods (DeePSD, MGNet) on the D-LAYERS dataset, achieving lower Euclidean errors (mm) across various garment types and conditions (with/without wind, tight/loose garments). For instance, in Table 2, LayersNet achieves errors around 467mm for overall garment animation, significantly better than DeePSD+ at ~1563mm.
  • Collision Reduction: LayersNet demonstrates significantly lower garment-to-body (H-Collision) and garment-to-garment (L-Collision) penetration rates. With RET and collision losses, H-Collision rates dropped dramatically (e.g., from 9.51% to 2.72% in Table 3).
  • Generalization: The model shows robust performance across different splits of the D-LAYERS dataset, indicating strong generalization capabilities to unseen garment combinations and dynamics.
  • Qualitative Results: Figure 3 showcases LayersNet's ability to produce realistic animations, capturing subtle dynamics like garment inertia and complex interactions between layers during human movement.

Limitations and Future Work

  • Limitations: While LayersNet shows strong generalization, the paper doesn't explicitly detail limitations in terms of extreme scenarios or specific types of materials beyond what's covered in the dataset. The reliance on a Transformer architecture might still pose computational challenges for extremely long sequences or very high-fidelity simulations, though the patch-based approach mitigates this.
  • Future Work: The authors suggest that the D-LAYERS dataset can be used for exploring other tasks like optical flow estimation, 3D reconstruction from images, and physics parameter estimation. The proposed method's ability to handle diverse interactions could be extended to more complex physical phenomena or a wider range of garment materials and properties.

Summary updated

Abstract

Most existing 3D garment animation datasets are restricted to human bodies with single-layered garments. Even though cases with upper shirts and lower pants are included, only a few overlap areas among such garment combinations exist. Moreover, they often regard human body movement as the only driving factor that causes garment animation. Approaches developed on top of these datasets thus tend to model garments as functions of human body parameters such as body shape and pose. While such treatment leads to promising performance on existing datasets, it leaves a gap between experimental environments and real scenarios, where a body can wear multiple layered garments and the corresponding garment dynamics can be affected by environmental factors and garment attributes. Consequently, existing approaches often struggle to generalize to multi-layered garments and realistic scenarios. To facilitate the advance of 3D garment animation toward handling more challenging cases, this paper presents a new large-scale synthetic dataset called LAYERS, covering 4,900 different combinations of multi-layered garments with 700k frames in total. The animation of these multi-layered garments follows the laws of physics and is affected by not only human body movements but also random environmental wind and garment attributes. To demonstrate the quality of LAYERS, we further propose a novel method, LayersNet, for 3D garment animation, which represents garments as unions of particles and subsequently adopts a neural network to animate garments via particle-based simulation. In this way, the interactions between different parts of one garment, different garments on the same body, and garments against various driving factors, can be naturally and uniformly handled via the interactions of particles. Through comprehensive experiments, LayersNet demonstrates superior performance in terms of animation accuracy and generality over baselines. The proposed dataset, LAYERS, as well as the proposed method, LayersNet, will be publicly available.
BibTeX
@inproceedings{shao2023multi,
  author = {Yidi Shao and Chen Change Loy and Bo Dai},
  title = {{Multi-Layered 3D Garments Animation}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=vmFwJeiSx4X},
}
Figure