Abstract
We propose X-Portrait, an innovative conditional diffusion model tailored for generating expressive and temporally coherent portrait animation. Specifically, given a single portrait as appearance reference, we aim to animate it with motion derived from a driving video, capturing both highly dynamic and subtle facial expressions along with wide-range head movements. As its core, we leverage the generative prior of a pre-trained diffusion model as the rendering backbone, while achieve fine-grained head pose and expression control with novel controlling signals within the framework of ControlNet. In contrast to conventional coarse explicit controls such as facial landmarks, our motion control module is learned to interpret the dynamics directly from the original driving RGB inputs. The motion accuracy is further enhanced with a patch-based local control module that effectively enhance the motion attention to small-scale nuances like eyeball positions. Notably, to mitigate the identity leakage from the driving signals, we train our motion control modules with scaling-augmented cross-identity images, ensuring maximized disentanglement from the appearance reference modules. Experimental results demonstrate the universal effectiveness of X-Portrait across a diverse range of facial portraits and expressive driving sequences, and showcase its proficiency in generating captivating portrait animations with consistently maintained identity characteristics.
Introduction
Portrait animation—the process of animating a static image based on a driving video—is a challenging task that requires balancing high expressiveness with strict identity preservation. Traditional methods often struggle with two primary issues: the "uncanny valley" effect caused by low-resolution motion representations and "identity leakage," where the animated subject begins to resemble the person in the driving video rather than the original portrait.
Existing frameworks typically rely on explicit controls like facial landmarks or skeletons. While effective for general head orientation, these sparse representations fail to capture subtle nuances such as the specific shape of a pout, the glint in an eye, or complex mouth movements. Furthermore, because these models are often trained on self-driven data (where the source and driver are the same), they tend to learn a "shortcut" that conflates motion with identity, leading to a "Frankenstein" effect in cross-identity scenarios.
Methodology
X-Portrait addresses these challenges by leveraging the generative power of Stable Diffusion (SD 1.5) and introducing three key architectural innovations:
1. Implicit Motion Control
Moving beyond rigid landmarks, X-Portrait utilizes a ControlNet-based architecture that interprets motion directly from RGB video frames. By processing raw pixels, the model can "see" fine-grained details—such as teeth, tongue movements, and gaze—that are typically lost in landmark-based systems.
2. Cross-Identity Training with Scaling Augmentation
To decouple identity from motion, the researchers employed a teacher-student training paradigm. Using a pre-trained reenactment network, they generated synthetic training pairs where "Person A" moves like "Person B." By applying random scaling to the driving images, the model is forced to ignore the driver's head shape and size, relying solely on the reference portrait for identity and the driving video for motion dynamics.
3. Hierarchical Local Control
To capture the most subtle expressive nuances, X-Portrait incorporates an auxiliary control branch focusing on 128x128 patches around the eyes and mouth. This hierarchical approach ensures that small-scale movements, such as eyeball position and lip tremors, are prioritized during the generation process, significantly reducing the "uncanny valley" effect.
Datasets and Evaluation
The model was trained on an extensive in-house dataset featuring 550 subjects performing 42 distinct expressions, totaling 20 minutes of high-quality talking head footage at 512x512 resolution.
To evaluate "out-of-domain" performance, the authors benchmarked X-Portrait against 100 "in-the-wild" portraits from platforms like Midjourney and DeviantArt. This test set included diverse artistic styles, ranging from 3D cartoons and oil paintings to marble statues and anime characters. Quantitative benchmarking was performed using 200 licensed test videos.
Results and Insights
X-Portrait significantly outperforms current state-of-the-art (SOTA) models, such as MagicDance and Face Vid2vid, across both quantitative and qualitative metrics.
Quantitative Performance
The model achieved a superior ID Similarity score of 0.689, indicating better preservation of the original subject's features compared to competitors. It also recorded the lowest errors in expression and pose transfer, confirming the accuracy of its implicit motion control.
Qualitative Generalization
One of the most striking features of X-Portrait is its zero-shot generalization capability. Because it inherits the vast prior knowledge of Stable Diffusion, it can animate artistic styles it never encountered during motion training.
In user studies, 83.23% of participants preferred X-Portrait’s results, noting its ability to handle extreme expressions—like yawning or pouting—without the blurring or distortion common in other models.
Conclusion and Future Work
X-Portrait represents a significant leap forward in expressive portrait animation. By replacing explicit landmarks with implicit RGB-based motion control and employing hierarchical attention, it achieves a level of detail and identity preservation previously unattainable.
While the model shows remarkable results, the authors identify areas for future refinement, including reducing dependency on the "teacher" model for extreme expressions and improving the temporal consistency of complex inner-mouth details. Future work may also extend these techniques to full-body gestures and more intricate hand-to-face interactions.
Summary updated