CVTHead: One-shot Controllable Head Avatar with Vertex-feature Transformer

Haoyu Ma, Tong Zhang, Shanlin Sun, Xiangyi Yan, Kun Han, Xiaohui Xie

Abstract

Reconstructing personalized animatable head avatars has significant implications in the fields of AR/VR. Existing methods for achieving explicit face control of 3D Morphable Models (3DMM) typically rely on multi-view images or videos of a single subject, making the reconstruction process complex. Additionally, the traditional rendering pipeline is time-consuming, limiting real-time animation possibilities. In this paper, we introduce CVTHead, a novel approach that generates controllable neural head avatars from a single reference image using point-based neural rendering. CVTHead considers the sparse vertices of mesh as the point set and employs the proposed Vertex-feature Transformer to learn local feature descriptors for each vertex. This enables the modeling of long-range dependencies among all the vertices. Experimental results on the VoxCeleb dataset demonstrate that CVTHead achieves comparable performance to state-of-the-art graphics-based methods. Moreover, it enables efficient rendering of novel human heads with various expressions, head poses, and camera views. These attributes can be explicitly controlled using the coefficients of 3DMMs, facilitating versatile and realistic animation in real-time scenarios.

Introduction

Creating a digital "puppet" of a human head—an animatable avatar—usually requires a trade-off between quality, speed, and data. NeRF-based methods often require minutes of multi-view video, while warp-based methods create "melting" artifacts during large head rotations. Graphics-based methods use 3D meshes but often rely on slow, traditional rendering pipelines unsuitable for real-time applications like AR/VR or teleconferencing.

CVTHead addresses these challenges by generating fully controllable 3D head avatars from just a single reference image. By combining 3D Morphable Models (3DMM) with a specialized Vertex-feature Transformer, it achieves high-fidelity, real-time animation (24.3 FPS) that remains robust even when the subject turns their head or parts of the face are occluded.

Methodology

The CVTHead pipeline treats 3D vertices as "tokens" in a transformer architecture to overcome the limitations of fixed pixel-aligned feature projection.

The CVTHead architecture: Source and driven images are processed via DECA and FLAME to generate vertices, which are then enhanced by the Vertex-feature Transformer and rendered via a neural point renderer.

Geometry Prior and Encoding

The system uses the FLAME model to provide a structural base. A pre-trained encoder (DECA) extracts shape, expression, and pose coefficients from the source and "driving" images. These coefficients allow for explicit control over the generated avatar's movements.

Vertex-feature Transformer

This is the core innovation. Instead of a fixed 2D-to-3D projection, the model treats each mesh vertex as a query token. Using a Transformer's global attention mechanism, each vertex can "attend" to the most relevant features in the source image. This allows the model to infer features for occluded regions by looking at symmetrical parts of the face or neighboring visible regions, effectively "hallucinating" missing details.

Point-based Neural Rendering

Rather than using heavy volumetric rendering, CVTHead projects the learned vertex features and depth into a 2D feature map. A U-Net then translates these raw features into a photorealistic RGB image, significantly reducing computational load.

Key Results

CVTHead was evaluated on the VoxCeleb1 and VoxCeleb2 datasets, demonstrating superior performance in both speed and visual fidelity.

  • Efficiency: CVTHead achieves 24.3 FPS on an NVIDIA RTX 3090Ti, nearly doubling the speed of previous state-of-the-art mesh-based methods like ROME (12.9 FPS).
  • Visual Quality: In cross-identity reenactment, CVTHead outperformed competitors in FID (Fréchet Inception Distance), indicating higher image realism.
Qualitative comparison showing CVTHead's ability to maintain sharp textures and consistent identity compared to ROME, especially during significant head rotations.

As shown in the qualitative results, while other models produce blurry or dark patches during large rotations, CVTHead maintains sharp textures and consistent identity by learning vertex descriptors in a canonical space.

Conclusion

CVTHead presents a robust solution for real-time, one-shot head avatar generation. By moving away from rigid geometric projections to an attention-based feature lookup, it effectively handles occlusions and large poses. While the quality remains tied to the accuracy of the initial 3D mesh estimation, the framework's efficiency makes it highly viable for real-time AR/VR applications. Future work may involve integrating generative tools like Diffusion Models to further enhance fine-grained texture quality.

Summary updated

Abstract

Reconstructing personalized animatable head avatars has significant implications in the fields of AR/VR. Existing methods for achieving explicit face control of 3D Morphable Models (3DMM) typically rely on multi-view images or videos of a single subject, making the reconstruction process complex. Additionally, the traditional rendering pipeline is time-consuming, limiting real-time animation possibilities. In this paper, we introduce CVTHead, a novel approach that generates controllable neural head avatars from a single reference image using point-based neural rendering. CVTHead considers the sparse vertices of mesh as the point set and employs the proposed Vertex-feature Transformer to learn local feature descriptors for each vertex. This enables the modeling of long-range dependencies among all the vertices. Experimental results on the VoxCeleb dataset demonstrate that CVTHead achieves comparable performance to state-of-the-art graphics-based methods. Moreover, it enables efficient rendering of novel human heads with various expressions, head poses, and camera views. These attributes can be explicitly controlled using the coefficients of 3DMMs, facilitating versatile and realistic animation in real-time scenarios.
BibTeX
@inproceedings{ma2024cvthead,
  author = {Haoyu Ma and Tong Zhang and Shanlin Sun and Xiangyi Yan and Kun Han and Xiaohui Xie},
  title = {{CVTHead: One-shot Controllable Head Avatar with Vertex-feature Transformer}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00602},
}
Figure