Abstract
Creating realistic, animatable 3D human avatars from sparse video data remains a significant challenge, often forcing a trade-off between structural consistency and high-frequency detail. We present 3D²-Actor, a framework that bridges this gap by interleaving 2D diffusion-based denoising with a 3D Gaussian Splatting (3DGS) "rectifier." By anchoring 3D Gaussians to a human mesh via local coordinate representations, our method generates avatars with realistic, pose-dependent details—such as dynamic clothing wrinkles—that maintain multi-view and temporal consistency. Experimental results on the ZJU-MoCap dataset demonstrate that 3D²-Actor significantly outperforms existing state-of-the-art methods in perceptual realism and temporal stability.
Introduction
The generation of high-fidelity human avatars is central to applications in virtual reality and digital entertainment. Existing approaches typically fall into two categories: 3D-centric methods (e.g., NeRF, 3DGS) and 2D-centric generative models (e.g., Diffusion). While 3D-centric methods excel at spatial consistency, they often struggle to capture stochastic, pose-dependent details like fabric textures, leading to "baked-in" or blurry appearances. Conversely, 2D diffusion models produce visually rich images but lack the inherent 3D awareness required for consistent multi-view rendering.
3D²-Actor addresses these limitations by framing avatar modeling as an iterative denoising process that enforces geometric truth. The key insight is to leverage 2D diffusion for its generative "hallucination" capabilities while using a 3D representation as a spatial filter to ensure consistency across viewpoints.
Methodology
The 3D²-Actor pipeline consists of an interleaved denoising process that alternates between pixel-level refinement and 3D structural alignment.
3D-Aware Denoising Pipeline
The core of the system is a Pixel-level 2D Denoiser conditioned on SMPL segmentation maps. This module refines noisy inputs to generate high-frequency details like fabric textures. To maintain spatial integrity, the output is passed to a Gaussian-based 3D Rectifier. This rectifier reconstructs a set of 3D Gaussians anchored to a standard SMPL mesh using a local coordinate representation. By predicting positions relative to the nearest mesh triangle (barycentric coordinates), the model ensures that fine details naturally follow the body's deformation during animation.
Temporal Consistency
For video sequences, the model employs Gaussian Consistency Sampling. Instead of independent frame generation, it uses local coordinates from the preceding frame to initialize the current one, effectively suppressing temporal flickering and ensuring persistent details across frames.
Experiments and Results
We evaluated 3D²-Actor on the ZJU-MoCap dataset, utilizing multi-view videos of diverse human actions. The model was trained on 80% of the frames and tested on novel poses.
Quantitative Performance
3D²-Actor demonstrates superior performance in perceptual metrics. On sequence 313, our model achieved an FID of 19.5, substantially outperforming baselines such as ARAH (32.6) and PoseVocab (27.5). While some baselines may achieve higher PSNR—a metric sensitive to exact pixel alignment—3D²-Actor's lower LPIPS and FID scores reflect its ability to generate more realistic and plausible human textures.
Qualitative Comparison
Visual analysis confirms that 3D²-Actor produces significantly sharper clothing wrinkles and avoids common artifacts like "color bleeding" between body parts, which frequently plagues previous methods.
Conclusion
3D²-Actor successfully combines the generative power of 2D diffusion with the structural robustness of 3D Gaussian Splatting. By anchoring details to a mesh scaffold via local coordinates, it enables the creation of highly realistic, animatable avatars. Future work will focus on reducing the computational overhead of the iterative denoising process and improving robustness to initial mesh fitting errors.
Summary updated