Abstract
Reconstructing personalized animatable head avatars has significant implications in the fields of AR/VR. Existing methods for achieving explicit face control of 3D Morphable Models (3DMM) typically rely on multi-view images or videos of a single subject, making the reconstruction process complex. Additionally, the traditional rendering pipeline is time-consuming, limiting real-time animation possibilities. In this paper, we introduce CVTHead, a novel approach that generates controllable neural head avatars from a single reference image using point-based neural rendering. CVTHead considers the sparse vertices of mesh as the point set and employs the proposed Vertex-feature Transformer to learn local feature descriptors for each vertex. This enables the modeling of long-range dependencies among all the vertices. Experimental results on the VoxCeleb dataset demonstrate that CVTHead achieves comparable performance to state-of-the-art graphics-based methods. Moreover, it enables efficient rendering of novel human heads with various expressions, head poses, and camera views. These attributes can be explicitly controlled using the coefficients of 3DMMs, facilitating versatile and realistic animation in real-time scenarios.
Introduction
Creating a digital "puppet" of a human head—an animatable avatar—usually requires a trade-off between quality, speed, and data. NeRF-based methods often require minutes of multi-view video, while warp-based methods create "melting" artifacts during large head rotations. Graphics-based methods use 3D meshes but often rely on slow, traditional rendering pipelines unsuitable for real-time applications like AR/VR or teleconferencing.
CVTHead addresses these challenges by generating fully controllable 3D head avatars from just a single reference image. By combining 3D Morphable Models (3DMM) with a specialized Vertex-feature Transformer, it achieves high-fidelity, real-time animation (24.3 FPS) that remains robust even when the subject turns their head or parts of the face are occluded.
Methodology
The CVTHead pipeline treats 3D vertices as "tokens" in a transformer architecture to overcome the limitations of fixed pixel-aligned feature projection.
Geometry Prior and Encoding
The system uses the FLAME model to provide a structural base. A pre-trained encoder (DECA) extracts shape, expression, and pose coefficients from the source and "driving" images. These coefficients allow for explicit control over the generated avatar's movements.
Vertex-feature Transformer
This is the core innovation. Instead of a fixed 2D-to-3D projection, the model treats each mesh vertex as a query token. Using a Transformer's global attention mechanism, each vertex can "attend" to the most relevant features in the source image. This allows the model to infer features for occluded regions by looking at symmetrical parts of the face or neighboring visible regions, effectively "hallucinating" missing details.
Point-based Neural Rendering
Rather than using heavy volumetric rendering, CVTHead projects the learned vertex features and depth into a 2D feature map. A U-Net then translates these raw features into a photorealistic RGB image, significantly reducing computational load.
Key Results
CVTHead was evaluated on the VoxCeleb1 and VoxCeleb2 datasets, demonstrating superior performance in both speed and visual fidelity.
- Efficiency: CVTHead achieves 24.3 FPS on an NVIDIA RTX 3090Ti, nearly doubling the speed of previous state-of-the-art mesh-based methods like ROME (12.9 FPS).
- Visual Quality: In cross-identity reenactment, CVTHead outperformed competitors in FID (Fréchet Inception Distance), indicating higher image realism.
As shown in the qualitative results, while other models produce blurry or dark patches during large rotations, CVTHead maintains sharp textures and consistent identity by learning vertex descriptors in a canonical space.
Conclusion
CVTHead presents a robust solution for real-time, one-shot head avatar generation. By moving away from rigid geometric projections to an attention-based feature lookup, it effectively handles occlusions and large poses. While the quality remains tied to the accuracy of the initial 3D mesh estimation, the framework's efficiency makes it highly viable for real-time AR/VR applications. Future work may involve integrating generative tools like Diffusion Models to further enhance fine-grained texture quality.
Summary updated