3D2-Actor: Learning Pose-Conditioned 3D-Aware Denoiser for Realistic Gaussian Avatar Modeling

Zichen Tang, Hongyu Yang, Hanchen Zhang, Jiaxin Chen, Di HuangarXiv 2024 · DOI · arXiv

Abstract

Creating realistic, animatable 3D human avatars from sparse video data remains a significant challenge, often forcing a trade-off between structural consistency and high-frequency detail. We present 3D²-Actor, a framework that bridges this gap by interleaving 2D diffusion-based denoising with a 3D Gaussian Splatting (3DGS) "rectifier." By anchoring 3D Gaussians to a human mesh via local coordinate representations, our method generates avatars with realistic, pose-dependent details—such as dynamic clothing wrinkles—that maintain multi-view and temporal consistency. Experimental results on the ZJU-MoCap dataset demonstrate that 3D²-Actor significantly outperforms existing state-of-the-art methods in perceptual realism and temporal stability.

Introduction

The generation of high-fidelity human avatars is central to applications in virtual reality and digital entertainment. Existing approaches typically fall into two categories: 3D-centric methods (e.g., NeRF, 3DGS) and 2D-centric generative models (e.g., Diffusion). While 3D-centric methods excel at spatial consistency, they often struggle to capture stochastic, pose-dependent details like fabric textures, leading to "baked-in" or blurry appearances. Conversely, 2D diffusion models produce visually rich images but lack the inherent 3D awareness required for consistent multi-view rendering.

3D²-Actor addresses these limitations by framing avatar modeling as an iterative denoising process that enforces geometric truth. The key insight is to leverage 2D diffusion for its generative "hallucination" capabilities while using a 3D representation as a spatial filter to ensure consistency across viewpoints.

Methodology

The 3D²-Actor pipeline consists of an interleaved denoising process that alternates between pixel-level refinement and 3D structural alignment.

Overview of the 3D²-Actor framework, illustrating the interleaved 2D denoising and 3D rectification process.

3D-Aware Denoising Pipeline

The core of the system is a Pixel-level 2D Denoiser conditioned on SMPL segmentation maps. This module refines noisy inputs to generate high-frequency details like fabric textures. To maintain spatial integrity, the output is passed to a Gaussian-based 3D Rectifier. This rectifier reconstructs a set of 3D Gaussians anchored to a standard SMPL mesh using a local coordinate representation. By predicting positions relative to the nearest mesh triangle (barycentric coordinates), the model ensures that fine details naturally follow the body's deformation during animation.

Temporal Consistency

For video sequences, the model employs Gaussian Consistency Sampling. Instead of independent frame generation, it uses local coordinates from the preceding frame to initialize the current one, effectively suppressing temporal flickering and ensuring persistent details across frames.

Experiments and Results

We evaluated 3D²-Actor on the ZJU-MoCap dataset, utilizing multi-view videos of diverse human actions. The model was trained on 80% of the frames and tested on novel poses.

Quantitative Performance

3D²-Actor demonstrates superior performance in perceptual metrics. On sequence 313, our model achieved an FID of 19.5, substantially outperforming baselines such as ARAH (32.6) and PoseVocab (27.5). While some baselines may achieve higher PSNR—a metric sensitive to exact pixel alignment—3D²-Actor's lower LPIPS and FID scores reflect its ability to generate more realistic and plausible human textures.

Qualitative Comparison

Visual analysis confirms that 3D²-Actor produces significantly sharper clothing wrinkles and avoids common artifacts like "color bleeding" between body parts, which frequently plagues previous methods.

Qualitative comparison of 3D²-Actor against state-of-the-art methods ARAH and PoseVocab, highlighting superior detail in clothing wrinkles.

Conclusion

3D²-Actor successfully combines the generative power of 2D diffusion with the structural robustness of 3D Gaussian Splatting. By anchoring details to a mesh scaffold via local coordinates, it enables the creation of highly realistic, animatable avatars. Future work will focus on reducing the computational overhead of the iterative denoising process and improving robustness to initial mesh fitting errors.

Summary updated

BibTeX
@misc{tang2024d,
  author = {Zichen Tang and Hongyu Yang and Hanchen Zhang and Jiaxin Chen and Di Huang},
  title = {{3D2-Actor: Learning Pose-Conditioned 3D-Aware Denoiser for Realistic Gaussian Avatar Modeling}},
  year = {2024},
  eprint = {2412.11599},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.11599},
}
Figure