Abstract
PhysAvatar is a novel framework that reconstructs 3D human avatars from multi-view video by jointly estimating their appearance and the underlying physics of their clothing. By automatically learning fabric parameters like stiffness and density, it enables realistic animation of loose-fitting garments and high-fidelity relighting in novel scenarios.
Introduction
Creating realistic digital humans is a cornerstone of VR, gaming, and telepresence. While existing techniques are proficient at reconstructing "tightly-clothed" humans using Linear Blending Skinning (LBS), loose-fitting clothing—such as dresses or oversized shirts—remains a significant challenge. Traditional methods often treat clothing as a rigid part of the body or rely on neural networks that overfit to training motions, leading to artifacts like "stretching" or "melting" during novel movements. Furthermore, many neural rendering techniques "bake" lighting into textures, preventing realistic relighting.
PhysAvatar addresses these limitations by integrating inverse rendering with inverse physics to estimate material properties and appearance simultaneously.
Methodology
The PhysAvatar pipeline consists of three primary stages designed to decouple geometry, physics, and appearance.
1. Mesh Tracking with 4D Gaussians
The system adapts 3D Gaussian Splatting to align with a reconstructed mesh. By attaching Gaussians to mesh faces and tracking them across video frames, the framework establishes dense spatio-temporal correspondences. This provides a geometric ground truth for cloth motion.
2. Inverse Physics Optimization
At the core of PhysAvatar is the use of a state-of-the-art physics simulator (C-IPC) to model cloth dynamics. Rather than manual tuning, the system employs gradient-based optimization to automatically estimate three key scalar parameters: density ($\rho$), membrane stiffness ($\kappa_s$), and bending stiffness ($\kappa_b$). By forcing the clothing to obey physical laws, the model generalizes naturally to unseen motions.
3. Physically Based Inverse Rendering
To enable relighting, the framework utilizes the Mitsuba 3 differentiable renderer. It separates time-invariant diffuse textures from ambient lighting, effectively removing "baked-in" shadows caused by self-occlusion. This results in clean texture maps that react naturally to new light sources.
Results and Evaluation
PhysAvatar was evaluated using the ActorHQ dataset, featuring characters with diverse clothing styles. To test generalization, the model was applied to motions from the AMASS and Mixamo datasets, which were not seen during training.
Quantitative Performance
PhysAvatar significantly outperforms state-of-the-art baselines such as ARAH, TAVA, and GS-Avatar in geometric accuracy. It achieved a leading Chamfer Distance of $0.55 \times 10^{-3}$ and an F-Score of $92.9$. Perceptually, it recorded the best LPIPS score ($0.035$), indicating superior visual fidelity.
Qualitative Capabilities
The framework successfully captures fine-grained wrinkles and complex cloth-body collisions. As shown in the results, the model allows for seamless relighting and "redressing," with outputs compatible with standard graphics engines like Blender.
Discussion and Future Work
By putting physics "in the loop," PhysAvatar moves beyond purely data-driven models that struggle with out-of-distribution data. However, the authors identify several areas for future research:
- Automation: Reducing the need for manual garment segmentation and UV unwrapping.
- Collider Precision: Improving the accuracy of the SMPL-X body collider to prevent minor clipping artifacts.
- Data Efficiency: Transitioning from dense multi-view requirements (160 cameras) to sparse-view settings to make the technology more accessible.
Conclusion
PhysAvatar represents a significant step toward photorealistic digital humans by combining the strengths of neural rendering and physical simulation. Its ability to generalize to novel motions and lighting conditions makes it a powerful tool for the future of immersive digital experiences.
Summary updated