Abstract
CVTHead is a novel framework designed to generate fully controllable 3D head avatars from a single "one-shot" image. By replacing traditional 2D feature projection with a Vertex-feature Transformer, the system achieves state-of-the-art realism and handles occlusions—such as turned heads—significantly better than previous methods. Operating at 24.3 FPS, CVTHead provides a real-time solution for high-fidelity digital reenactment.
Introduction
Creating a digital "puppet" of a person's head for reenactment typically involves a trade-off: high realism requires extensive multi-view video data, while "one-shot" methods often struggle with large rotations or occluded regions. Most current one-shot approaches rely on pixel-aligned features, which fail when parts of the face (like the back of an ear) are hidden in the source image. CVTHead addresses these challenges by leveraging Transformers to manage long-range dependencies in 3D geometry, ensuring consistency even under extreme poses.
Methodology
The CVTHead pipeline is divided into three primary stages to decouple identity from motion and render high-quality results efficiently.
1. Geometry Prior (FLAME)
The framework utilizes the FLAME model, a standard 3D Morphable Model (3DMM), as its geometric foundation. A pre-trained DECA network estimates the initial 3D shape, expression, and pose from the source image, providing a robust starting point for the avatar.
2. Vertex-feature Transformer
The core innovation of CVTHead is the Vertex-feature Transformer. Instead of simple 2D projection, each vertex of the 3D mesh is treated as a token. Through global self-attention, vertices representing occluded areas can "attend" to visible parts of the face to learn accurate canonical feature descriptors. This mechanism effectively solves the "garbage data" problem associated with traditional pixel-aligned methods.
3. Neural Point-based Rendering
To ensure real-time performance, CVTHead avoids slow differentiable mesh rendering. Instead, it projects learned vertex features and depth values onto a 2D plane. A U-Net-based neural renderer then processes these projections to generate the final image, significantly reducing computational overhead.
Experiments and Results
The model was trained and evaluated on the VoxCeleb1 and VoxCeleb2 datasets, focusing on unseen identity reenactment.
- Performance: CVTHead achieves 24.3 FPS on an NVIDIA RTX 3090Ti, nearly doubling the speed of competitors like ROME (12.9 FPS).
- Visual Quality: In cross-identity reenactment tasks, CVTHead outperformed existing methods (e.g., FOMM, ROME) in Fréchet Inception Distance (FID), indicating superior image realism.
- Robustness: Qualitative analysis demonstrates that the model maintains sharp textures and structural integrity during large head rotations, where previous methods typically produce artifacts or blurriness.
Discussion and Future Work
While CVTHead represents a significant advancement, it remains dependent on the accuracy of the initial DECA mesh reconstruction. Errors in the underlying 3D shape can propagate to the final output. Additionally, extreme expressions can sometimes lead to "identity drift" due to imperfect disentanglement of shape and expression. Future research may involve integrating Diffusion Models to enhance fine-grained textures, such as skin pores and individual hairs, beyond the current resolution of neural renderers.
Summary updated