PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual Observations

Yang Zheng, Qingqing Zhao, Guandao Yang, Wang Yifan, Donglai Xiang, Florian Dubost, Dmitry Lagun, Thabo Beeler, Federico Tombari, Leonidas J. Guibas, Gordon Wetzstein

Abstract

PhysAvatar is a novel framework that reconstructs 3D human avatars from multi-view video by jointly estimating their appearance and the underlying physics of their clothing. By automatically learning fabric parameters like stiffness and density, it enables realistic animation of loose-fitting garments and high-fidelity relighting in novel scenarios.

Introduction

Creating realistic digital humans is a cornerstone of VR, gaming, and telepresence. While existing techniques are proficient at reconstructing "tightly-clothed" humans using Linear Blending Skinning (LBS), loose-fitting clothing—such as dresses or oversized shirts—remains a significant challenge. Traditional methods often treat clothing as a rigid part of the body or rely on neural networks that overfit to training motions, leading to artifacts like "stretching" or "melting" during novel movements. Furthermore, many neural rendering techniques "bake" lighting into textures, preventing realistic relighting.

PhysAvatar addresses these limitations by integrating inverse rendering with inverse physics to estimate material properties and appearance simultaneously.

Methodology

The PhysAvatar pipeline consists of three primary stages designed to decouple geometry, physics, and appearance.

PhysAvatar Pipeline Overview

1. Mesh Tracking with 4D Gaussians

The system adapts 3D Gaussian Splatting to align with a reconstructed mesh. By attaching Gaussians to mesh faces and tracking them across video frames, the framework establishes dense spatio-temporal correspondences. This provides a geometric ground truth for cloth motion.

2. Inverse Physics Optimization

At the core of PhysAvatar is the use of a state-of-the-art physics simulator (C-IPC) to model cloth dynamics. Rather than manual tuning, the system employs gradient-based optimization to automatically estimate three key scalar parameters: density ($\rho$), membrane stiffness ($\kappa_s$), and bending stiffness ($\kappa_b$). By forcing the clothing to obey physical laws, the model generalizes naturally to unseen motions.

3. Physically Based Inverse Rendering

To enable relighting, the framework utilizes the Mitsuba 3 differentiable renderer. It separates time-invariant diffuse textures from ambient lighting, effectively removing "baked-in" shadows caused by self-occlusion. This results in clean texture maps that react naturally to new light sources.

Results and Evaluation

PhysAvatar was evaluated using the ActorHQ dataset, featuring characters with diverse clothing styles. To test generalization, the model was applied to motions from the AMASS and Mixamo datasets, which were not seen during training.

Qualitative results showing diverse poses, relighting, and cloth simulation.

Quantitative Performance

PhysAvatar significantly outperforms state-of-the-art baselines such as ARAH, TAVA, and GS-Avatar in geometric accuracy. It achieved a leading Chamfer Distance of $0.55 \times 10^{-3}$ and an F-Score of $92.9$. Perceptually, it recorded the best LPIPS score ($0.035$), indicating superior visual fidelity.

Qualitative Capabilities

The framework successfully captures fine-grained wrinkles and complex cloth-body collisions. As shown in the results, the model allows for seamless relighting and "redressing," with outputs compatible with standard graphics engines like Blender.

Discussion and Future Work

By putting physics "in the loop," PhysAvatar moves beyond purely data-driven models that struggle with out-of-distribution data. However, the authors identify several areas for future research:

  • Automation: Reducing the need for manual garment segmentation and UV unwrapping.
  • Collider Precision: Improving the accuracy of the SMPL-X body collider to prevent minor clipping artifacts.
  • Data Efficiency: Transitioning from dense multi-view requirements (160 cameras) to sparse-view settings to make the technology more accessible.

Conclusion

PhysAvatar represents a significant step toward photorealistic digital humans by combining the strengths of neural rendering and physical simulation. Its ability to generalize to novel motions and lighting conditions makes it a powerful tool for the future of immersive digital experiences.

Summary updated

Abstract

Modeling and rendering photorealistic avatars is of crucial importance in many applications. Existing methods that build a 3D avatar from visual observations, however, struggle to reconstruct clothed humans. We introduce PhysAvatar, a novel framework that combines inverse rendering with inverse physics to automatically estimate the shape and appearance of a human from multi-view video data along with the physical parameters of the fabric of their clothes. For this purpose, we adopt a mesh-aligned 4D Gaussian technique for spatio-temporal mesh tracking as well as a physically based inverse renderer to estimate the intrinsic material properties. PhysAvatar integrates a physics simulator to estimate the physical parameters of the garments using gradient-based optimization in a principled manner. These novel capabilities enable PhysAvatar to create high-quality novel-view renderings of avatars dressed in loose-fitting clothes under motions and lighting conditions not seen in the training data. This marks a significant advancement towards modeling photorealistic digital humans using physically based inverse rendering with physics in the loop. Our project website is at: https://qingqing-zhao.github.io/PhysAvatar.
BibTeX
@inproceedings{zheng2024physavatar,
  author = {Yang Zheng and Qingqing Zhao and Guandao Yang and Wang Yifan and Donglai Xiang and Florian Dubost and Dmitry Lagun and Thabo Beeler and Federico Tombari and Leonidas J. Guibas and Gordon Wetzstein},
  title = {{PhysAvatar: Learning the Physics of Dressed 3D Avatars from Visual Observations}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-72913-3\_15},
}
Figure