Robust Video Portrait Reenactment via Personalized Representation Quantization

Kaisiyuan Wang, Changcheng Liang, Hang Zhou, Jiaxiang Tang, Qianyi Wu, Dongliang He, Zhibin Hong, Jingtuo Liu, Errui Ding, Ziwei Liu, Jingdong Wang

Abstract

Video portrait reenactment aims to transfer the facial expressions and head poses of a driving video to a target person's image. Current methods often struggle with robustness under extreme poses or fail to maintain high-fidelity identity details. We present VPNQ (Video Portrait via Non-local Quantization Modeling), a framework that addresses these challenges by learning a personalized quantized texture dictionary. By decoupling texture from geometry and utilizing a Spatio-Temporal Transformer, VPNQ produces temporally consistent, high-fidelity videos that outperform existing GAN and NeRF-based approaches.

Introduction

Video portrait reenactment—the process of "puppeteering" a target person's face using the movements of a driving video—faces two major hurdles: robustness and the trade-off between fidelity and efficiency.

Most existing models struggle with "out-of-distribution" poses, such as extreme head tilts not seen during training. This often results in distorted facial features or "identity drift," where the subject's appearance begins to morph into the driving person's identity. Furthermore, while GAN-based methods are efficient, they often produce blurry textures or unstable results in complex areas like the mouth. Conversely, Neural Radiance Fields (NeRF) offer high quality but are computationally expensive and sensitive to camera pose errors. VPNQ bridges this gap by combining the robustness of landmark-based systems with the high fidelity of person-specific modeling.

Methodology

The VPNQ framework operates in two distinct stages: personalized codebook learning and spatio-temporal code transformation.

1. Personalized Codebook Learning

Instead of relying on a universal, one-size-fits-all codebook, VPNQ trains a person-specific autoencoder on the target subject. The encoder decomposes the target's images into local patches, which are then stored in a quantized codebook. This "Texture Dictionary" approach is crucial because it decouples the texture (the visual appearance of a patch) from the geometry (its spatial location). This position invariance allows the model to re-assemble the face accurately even in entirely new poses.

The architecture for Personalized Codebook Learning, featuring an encoder $E_n$, a quantized codebook $\mathcal{C}$, and a decoder $D_e$.

2. Spatio-Temporal Code Transformer

Once the dictionary of the person's face is learned, the system predicts which tokens to place where based on a driving signal.

  • Driving Signal: The model uses simple 2D landmarks extracted via the DECA 3D model, keeping the system lightweight compared to 3D rendering methods.
  • The Transformer: A Vision Transformer (ViT) architecture takes these landmark maps and predicts the correct sequence of quantized tokens.
  • Temporal Strategy: To prevent the "jittering" common in AI-generated video, the Transformer processes multiple consecutive frames simultaneously, ensuring temporal consistency across the generated sequence.

Key Insights

  • Personalization as a Digital Twin: Universal codebooks often lose fine-grained details like specific wrinkles or unique accessories. A personalized codebook acts as a high-fidelity digital twin of the individual.
  • Quantization as a Buffer: Representing the face as discrete tokens makes the model robust to noise. If a driving landmark is slightly inaccurate, the model "snaps" to the nearest valid texture token rather than generating a distorted pixel blob.
  • Non-local Modeling: Using a Transformer allows the model to understand global relationships across the face, such as how jaw movement affects skin tension near the ears, which is vital for maintaining identity under extreme movement.

Experimental Results

VPNQ was evaluated against state-of-the-art (SOTA) models including Facev2v, LIA, NHA, and LSP across several high-quality datasets like HDTF and AD-Nerf.

Quantitative Performance

VPNQ achieved superior scores in PSNR (image quality) and SSIM (structural similarity). For instance, on Testset C, it reached a PSNR of 32.36, significantly outperforming LSP's 31.44. In a Mean Opinion Score (MOS) study, users rated VPNQ 4.73/5.0 for generation quality and 4.80/5.0 for video realness.

Qualitative Superiority

In scenarios involving unseen poses, VPNQ maintained sharp boundaries and identity preservation where other methods produced blurring or warped features.

Qualitative comparison highlighting the reduction in artifacts (left) and improved texture fidelity (right) achieved by VPNQ.

Conclusion and Ethical Considerations

While VPNQ provides high-fidelity results, it currently requires person-specific training, which precludes "zero-shot" applications. Additionally, the authors acknowledge the potential for misuse in creating deepfakes. They emphasize that this technology should be developed alongside robust detection methods to mitigate ethical risks. Future work will focus on reducing the computational overhead of the personalized training phase.

Summary updated

Abstract

While progress has been made in the field of portrait reenactment, the problem of how to produce high-fidelity and robust videos remains. Recent studies normally find it challenging to handle rarely seen target poses due to the limitation of source data. This paper proposes the Video Portrait via Non-local Quantization Modeling (VPNQ) framework, which produces pose- and disturbance-robust reenactable video portraits. Our key insight is to learn position-invariant quantized local patch representations and build a mapping between simple driving signals and local textures with non-local spatial-temporal modeling. Specifically, instead of learning a universal quantized codebook, we identify that a personalized one can be trained to preserve desired position-invariant local details better. Then, a simple representation of projected landmarks can be used as sufficient driving signals to avoid 3D rendering. Following, we employ a carefully designed Spatio-Temporal Transformer to predict reasonable and temporally consistent quantized tokens from the driving signal. The predicted codes can be decoded back to robust and high-quality videos. Comprehensive experiments have been conducted to validate the effectiveness of our approach.
BibTeX
@inproceedings{wang2023robust,
  author = {Kaisiyuan Wang and Changcheng Liang and Hang Zhou and Jiaxiang Tang and Qianyi Wu and Dongliang He and Zhibin Hong and Jingtuo Liu and Errui Ding and Ziwei Liu and Jingdong Wang},
  title = {{Robust Video Portrait Reenactment via Personalized Representation Quantization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i2.25354},
}
Figure