TL;DR
ARG-Avatar is a highly efficient feedforward framework for reconstructing animatable 3D human avatars from a few images. By explicitly integrating geometric priors—like animation and rendering—directly into the model's attention mechanism, it achieves state-of-the-art quality with only 68M parameters, outperforming models nearly ten times its size.
What Problem Does This Paper Address?
Creating high-quality, controllable 3D avatars typically requires either lengthy per-person optimization or massive "scaling-first" feedforward models. Recent state-of-the-art methods (like the LHM series) rely on hundreds of millions of parameters and hundreds of thousands of private video sequences to learn how to map 2D image pixels to 3D avatar parts.
The authors argue that this brute-force scaling is unnecessary. Existing methods force neural networks to "re-learn" basic geometric correspondences that are already mathematically well-defined by human body templates and camera projection. ARG-Avatar aims to bridge this efficiency gap by using Animation-Rendering Guidance to help the model focus on the right parts of the input images from the start.
Methods
ARG-Avatar represents avatars using 3D Gaussian Splatting (3DGS) anchored to a standard human mesh template. The core of the architecture consists of two novel components:
- FACRoPE (Foreground Avatar Coordinate RoPE): Standard attention mechanisms are "geometry-blind," searching the entire image for features. FACRoPE uses the known body pose and camera geometry to project each 3D avatar token into the 2D image plane. These coordinates are then used in a Rotary Position Embedding (RoPE) framework, effectively telling each token exactly where it should "look" in the input image.
- ITR (Intermediate Token Rendering): This is a "rendering-in-the-loop" strategy. The network decodes its intermediate features into a coarse 3D avatar, renders it, and uses that projected geometry to guide subsequent layers. This iterative refinement ensures that the model's attention remains grounded in 3D space.
- Pose-Dependent Deformation: To handle non-rigid effects like loose clothing or hair, the model predicts residuals that adjust the canonical avatar based on the target driving pose.
Datasets
Unlike many competitors that rely on massive proprietary datasets, ARG-Avatar is trained exclusively on publicly available datasets:
- Training: A combination of DNARendering, 2K2K, 4D-Dress, Human4DiT, THuman2.1, and HumanVid, totaling approximately 6.8K subjects and 64K sequences.
- Evaluation: The primary benchmark is the NeuMan dataset, used to test out-of-distribution performance on "in-the-wild" captures.
Key Insights and Ideas
- Geometry is a Strong Prior: You don't need a billion parameters to learn that a hand in 3D corresponds to a hand in a 2D photo if you already have the math for Linear Blend Skinning (LBS) and camera projection.
- Disentanglement through Regularization: The authors introduced a "Common Shell Transfer" regularizer. This forces the model to capture shared geometric traits in the canonical (rest) pose rather than redundantly encoding them in every individual pose-dependent deformation.
- Efficiency as a Feature: By reducing the search space for attention, the model can be much smaller and faster without sacrificing (and actually improving) reconstruction quality.
Key Results
ARG-Avatar sets a new bar for efficiency and quality in feedforward reconstruction:
- Superior Quality: It achieves a 30.82 dB PSNR on the NeuMan benchmark, a 2.7 dB improvement over the previous state-of-the-art (LHM++).
- Massive Efficiency Gains: Compared to LHM++, it uses 9× fewer trainable parameters (68M vs. 609M), requires 14× less training compute, and runs 6× faster at inference time (2.3s vs. 14.2s for a 6-view reconstruction).
- Data Efficiency: It achieves these results using only ~21% of the training data sequences used by its closest competitors.
Limitations and Future Work
- Single-View Performance: While ARG-Avatar excels with two or more images, its performance drops with only a single input view. The authors note it is not currently trained to "inpaint" or hallucinate invisible regions (like the back of a person).
- Template Dependency: The system relies on a predefined human template (MHR/SMPL-X). While this provides the necessary geometric guidance, it may limit the model's ability to reconstruct extreme body shapes or very bulky clothing that deviates significantly from the template.
- Future Directions: Integrating generative priors (like diffusion models) could help the system handle single-view inputs more gracefully by hallucinating missing details.
Summary updated