3D²-Actor: Learning Pose-Conditioned 3D-Aware Denoiser for Realistic Gaussian Avatar Modeling

Zichen Tang, Hongyu Yang, Hanchen Zhang, Jiaxin Chen, Di HuangAAAI 2025 · DOI

Abstract

Creating realistic, animatable human avatars requires balancing high-fidelity texture generation with strict 3D geometric consistency. 3D²-Actor addresses this by introducing a hybrid framework that interleaves a pose-conditioned 2D diffusion-based denoiser with a 3D Gaussian Splatting (3DGS) rectifier. This "3D-aware" denoising process allows the model to capture fine-grained, pose-dependent details—such as dynamic clothing wrinkles—while ensuring structural stability across multiple views. Evaluated on the ZJU-MoCap dataset, 3D²-Actor demonstrates significant improvements in perceptual realism and temporal consistency over existing state-of-the-art methods.

Introduction

The synthesis of realistic digital avatars is a cornerstone for applications in virtual reality, gaming, and telepresence. However, existing approaches often struggle with a fundamental trade-off. 3D-centric methods, typically utilizing deformation fields in NeRF or 3DGS, provide strong geometric foundations but often fail to capture stochastic, pose-dependent details like the shifting wrinkles of a T-shirt, leading to blurred or static textures. Conversely, 2D-centric diffusion models excel at generating rich textures but lack inherent 3D awareness, resulting in "texture flickering" and structural inconsistencies when viewed from different angles.

3D²-Actor bridges this gap by merging the perceptual power of diffusion models with the spatial consistency of 3D rendering.

Methodology

The 3D²-Actor framework employs an interleaved pipeline that alternates between 2D denoising and 3D rectification steps to ensure that generated details are both realistic and spatially coherent.

The 3D²-Actor pipeline interleaves 2D denoising with 3D rectification to ensure geometric consistency.

Pose-Conditioned 2D Denoiser

The process begins with pure noise, which is iteratively refined by a U-Net-based denoiser. To ensure the generated content respects the target human pose, the denoiser is conditioned on body segmentation maps. Spatial Feature Transform (SFT) layers are integrated into the network to modulate feature maps based on these pose cues, allowing for the generation of pose-appropriate textures and fine-grained details.

Gaussian-based 3D Rectifier

To maintain 3D consistency, the model periodically applies a "3D Rectifier." This module projects 2D features into a 3D space to reconstruct a set of 3D Gaussians. These Gaussians are then rendered back into 2D images, effectively "rectifying" the 2D hallucinations to align with a consistent 3D structure. This prevents the model from generating details that cannot exist in a coherent 3D space.

Local Coordinate Representation and Temporal Stability

Generalization to unseen poses is achieved by predicting Gaussian positions relative to an underlying SMPL mesh scaffold. This mesh-anchored approach provides a strong geometric prior. For video synthesis, the model utilizes Gaussian Consistency Sampling, where the Gaussian positions from the previous frame initialize the current frame, ensuring smooth temporal transitions without the need for complex temporal modules.

Experimental Results

The model was evaluated on the ZJU-MoCap dataset, focusing on the synthesis of humans in novel, unseen poses.

Quantitative Performance

3D²-Actor significantly outperforms baselines such as ARAH and PoseVocab in perceptual metrics. On sequence 313, the model achieved an FID of 19.5, a substantial improvement over ARAH's 32.6. Similar gains were observed in LPIPS scores, confirming the model's ability to produce images that are perceptually closer to ground truth.

Qualitative Analysis

As illustrated in Figure 3, 3D²-Actor preserves sharp clothing wrinkles and clear limb boundaries where competing methods often produce blur or color bleeding. The interleaved rectification process ensures that these fine details remain stable as the avatar moves.

Qualitative comparison against ARAH and PoseVocab on the ZJU-MoCap dataset.

Discussion and Limitations

The primary strength of 3D²-Actor lies in its synergy: 2D diffusion handles the "what" (textures), while 3DGS handles the "where" (spatial consistency). However, the iterative nature of the diffusion process results in higher computational costs during inference compared to pure 3DGS methods. Additionally, while the model achieves high perceptual realism, it may occasionally produce lower PSNR scores because the "hallucinated" wrinkles, though realistic, may not perfectly align with the pixel-level ground truth. Future work will focus on accelerating the denoising process and extending the framework to handle loose-fitting clothing and complex accessories.

Summary updated

BibTeX
@inproceedings{tang2025d,
  author = {Zichen Tang and Hongyu Yang and Hanchen Zhang and Jiaxin Chen and Di Huang},
  title = {{3D²-Actor: Learning Pose-Conditioned 3D-Aware Denoiser for Realistic Gaussian Avatar Modeling}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i7.32789},
}
Figure