CVTHead: One-shot Controllable Head Avatar with Vertex-feature Transformer

Haoyu Ma, Tong Zhang, Shanlin Sun, Xiangyi Yan, Kun Han, Xiaohui Xie

Abstract

CVTHead is a novel framework designed to generate fully controllable 3D head avatars from a single "one-shot" image. By replacing traditional 2D feature projection with a Vertex-feature Transformer, the system achieves state-of-the-art realism and handles occlusions—such as turned heads—significantly better than previous methods. Operating at 24.3 FPS, CVTHead provides a real-time solution for high-fidelity digital reenactment.

Introduction

Creating a digital "puppet" of a person's head for reenactment typically involves a trade-off: high realism requires extensive multi-view video data, while "one-shot" methods often struggle with large rotations or occluded regions. Most current one-shot approaches rely on pixel-aligned features, which fail when parts of the face (like the back of an ear) are hidden in the source image. CVTHead addresses these challenges by leveraging Transformers to manage long-range dependencies in 3D geometry, ensuring consistency even under extreme poses.

Methodology

The CVTHead pipeline is divided into three primary stages to decouple identity from motion and render high-quality results efficiently.

The CVTHead pipeline architecture, illustrating the flow from source and driven images through the DECA encoder, FLAME geometry model, Vertex-feature Transformer, and final neural point rendering.

1. Geometry Prior (FLAME)

The framework utilizes the FLAME model, a standard 3D Morphable Model (3DMM), as its geometric foundation. A pre-trained DECA network estimates the initial 3D shape, expression, and pose from the source image, providing a robust starting point for the avatar.

2. Vertex-feature Transformer

The core innovation of CVTHead is the Vertex-feature Transformer. Instead of simple 2D projection, each vertex of the 3D mesh is treated as a token. Through global self-attention, vertices representing occluded areas can "attend" to visible parts of the face to learn accurate canonical feature descriptors. This mechanism effectively solves the "garbage data" problem associated with traditional pixel-aligned methods.

3. Neural Point-based Rendering

To ensure real-time performance, CVTHead avoids slow differentiable mesh rendering. Instead, it projects learned vertex features and depth values onto a 2D plane. A U-Net-based neural renderer then processes these projections to generate the final image, significantly reducing computational overhead.

Experiments and Results

The model was trained and evaluated on the VoxCeleb1 and VoxCeleb2 datasets, focusing on unseen identity reenactment.

  • Performance: CVTHead achieves 24.3 FPS on an NVIDIA RTX 3090Ti, nearly doubling the speed of competitors like ROME (12.9 FPS).
  • Visual Quality: In cross-identity reenactment tasks, CVTHead outperformed existing methods (e.g., FOMM, ROME) in Fréchet Inception Distance (FID), indicating superior image realism.
  • Robustness: Qualitative analysis demonstrates that the model maintains sharp textures and structural integrity during large head rotations, where previous methods typically produce artifacts or blurriness.

Discussion and Future Work

While CVTHead represents a significant advancement, it remains dependent on the accuracy of the initial DECA mesh reconstruction. Errors in the underlying 3D shape can propagate to the final output. Additionally, extreme expressions can sometimes lead to "identity drift" due to imperfect disentanglement of shape and expression. Future research may involve integrating Diffusion Models to enhance fine-grained textures, such as skin pores and individual hairs, beyond the current resolution of neural renderers.

Summary updated

Abstract

Reconstructing personalized animatable head avatars has significant implications in the fields of AR/VR. Existing methods for achieving explicit face control of 3D Morphable Models (3DMM) typically rely on multi-view images or videos of a single subject, making the reconstruction process complex. Additionally, the traditional rendering pipeline is time-consuming, limiting real-time animation possibilities. In this paper, we introduce CVTHead, a novel approach that generates controllable neural head avatars from a single reference image using point-based neural rendering. CVT-Head considers the sparse vertices of mesh as the point set and employs the proposed Vertex-feature Transformer to learn local feature descriptors for each vertex. This enables the modeling of long-range dependencies among all the vertices. Experimental results on the VoxCeleb dataset demonstrate that CVTHead achieves comparable performance to state-of-the-art graphics-based methods. Moreover, it enables efficient rendering of novel human heads with various expressions, head poses, and camera views. These attributes can be explicitly controlled using the coefficients of 3DMMs, facilitating versatile and realistic animation in real-time scenarios. Codes and pre-trained model can be found at https://github.com/HowieMa/CVTHead.
BibTeX
@inproceedings{ma2024cvthead,
  author = {Haoyu Ma and Tong Zhang and Shanlin Sun and Xiangyi Yan and Kun Han and Xiaohui Xie},
  title = {{CVTHead: One-shot Controllable Head Avatar with Vertex-feature Transformer}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00602},
}
Figure