DLCA-Recon: Dynamic Loose Clothing Avatar Reconstruction from Monocular Videos

Chunjie Luo, Fei Luo, Yusen Wang, Enxu Zhao, Chunxia XiaoAAAI 2024 · DOI

Introduction

Reconstructing a moving human in 3D from a single monocular camera remains a significant challenge in computer vision, with profound implications for augmented reality (AR), virtual reality (VR), and the gaming industry. While existing methods have achieved high-fidelity results for tight-fitting clothing using parametric models like SMPL, loose clothing—such as dresses, skirts, and oversized coats—presents a unique set of hurdles.

The difficulty stems from two primary factors. First, there is a geometric gap: loose garments often have complex topologies and significant distances from the underlying skin surface, making simple offset-based models inadequate. Second, there is dynamic randomness: fabric does not merely follow bone movement; it exhibits complex behaviors like swinging, folding, and reacting to inertia. Traditional methods often lack the necessary geometric constraints to model these dynamics, leading to noisy or discontinuous reconstructions.

The DLCA-Recon Framework

To address these issues, the authors introduce DLCA-Recon, a framework that shifts away from rigid body templates toward a physics-aware, flexible reconstruction approach.

Non-Parametric Initialization

A critical innovation in DLCA-Recon is its initialization strategy. Instead of starting with a generic "naked" body model (SMPL), the system utilizes ECON, a single-image reconstruction tool, to generate an initial clothed human mesh in a neutral "canonical" space. This provides a topologically accurate starting point that accounts for garments like skirts, which do not conform to the standard leg topology of parametric models.

Dynamic Deformation Field (DDF)

The core of the system is the Dynamic Deformation Field (DDF), which models fabric motion by decomposing it into two physics-inspired components:

  • Traction Forces: These represent the forces exerted on the clothing by the movement of the human joints.
  • Internal Forces: Modeled via a "spring model," these forces ensure the fabric maintains its structural integrity relative to the body's motion.

By embedding these constraints into a neural network (MLP), the model achieves better interpretability and more accurately simulates real-world fabric dynamics compared to purely data-driven approaches.

The DLCA-Recon pipeline overview, illustrating the flow from monocular video frames to pose estimation, dynamic deformation, and hybrid rendering.

Hybrid Representation and Refinement

DLCA-Recon maintains a dual representation to leverage the strengths of different modeling techniques:

  1. Explicit Mesh: Used for physics calculations and maintaining consistent topology.
  2. Implicit Signed Distance Field (SDF): Used for high-resolution neural rendering and capturing fine surface details.

The training process also employs a Delayed Optimization strategy. By initially disabling pose and skinning weight refinements, the non-rigid deformation network is allowed to stabilize, preventing model collapse during the early stages of training.

Experimental Evaluation

The researchers evaluated DLCA-Recon across several challenging datasets, including DeepCap (Antonia and Magdalena sequences) and DynaCap (FranziRed sequence). To test real-world robustness, they also introduced three new self-captured sequences (LCJ, LYZ, ZJ) recorded with a Canon EOS 6D Mark II.

Key Results

DLCA-Recon significantly outperforms state-of-the-art (SOTA) methods such as SelfRecon, HumanNeRF, and Vid2Avatar in both geometric accuracy and rendering quality.

  • Geometric Accuracy: On the DeepCap dataset, DLCA-Recon achieved a Normal Mean Absolute Error (MAE) of 6.14, a substantial improvement over SelfRecon's 11.19 (approximately a 45% reduction in error).
  • Visual Fidelity: The method consistently achieved higher PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) scores, indicating sharper and more realistic reconstructions.
  • Qualitative Performance: Visual comparisons demonstrate that DLCA-Recon maintains clean, continuous surfaces even during rapid movements, successfully reconstructing complex areas like the underside of loose clothing where other methods typically fail.
Qualitative results showing the high-fidelity reconstruction of loose clothing compared to the original video frames.

Discussion and Future Work

While DLCA-Recon represents a major step forward, some limitations remain. The framework's performance is currently sensitive to the quality of ground truth masks; poor background subtraction can degrade the final reconstruction. Additionally, the model relies on predicted 2D normal maps for fine details; inaccuracies in these 2D predictions can propagate into the 3D geometry.

Future research may focus on incorporating stronger 3D geometric priors or direct 3D supervision to further bridge the gap between 2D observations and 3D reality, potentially reducing the reliance on external 2D predictors.

Summary updated

BibTeX
@inproceedings{luo2024dlca,
  author = {Chunjie Luo and Fei Luo and Yusen Wang and Enxu Zhao and Chunxia Xiao},
  title = {{DLCA-Recon: Dynamic Loose Clothing Avatar Reconstruction from Monocular Videos}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i4.28189},
}
Figure