Introduction
Zero-shot portrait animation aims to animate a static reference image using the facial dynamics of a driving video. While existing diffusion-based methods have made significant strides, they often suffer from two critical flaws: identity leakage, where the reference portrait begins to resemble the driving subject, and a loss of expressiveness, failing to capture subtle or extreme facial movements.
X-NeMo addresses these challenges by introducing a novel pipeline that decouples motion from identity. By distilling driving motions into a 1D identity-agnostic latent descriptor and injecting it via cross-attention, X-NeMo achieves highly expressive reenactment while strictly preserving the original portrait's unique structural identity.
The Problem: Identity Leakage and Expression Loss
The primary cause of identity leakage in current models is the reliance on explicit 2D spatial signals, such as facial landmarks. When these signals are injected into a diffusion model using spatially-aligned modules like ControlNet, the network often takes a "shortcut." Instead of learning the semantic motion, it copies the facial structure and layout of the driving subject. This results in a "morphed" output that loses the essence of the original reference image. Furthermore, standard diffusion losses often prioritize low-frequency smoothness, washing out the fine details of micro-expressions.
Methodology: Decoupling Motion and Identity
X-NeMo abandons 2D spatial control maps in favor of a 1D latent motion descriptor. This architectural shift is central to its ability to preserve identity while capturing complex expressions.
1D Identity-Agnostic Motion Latent
An encoder compresses each driving frame into a compact 1D vector. This acts as an information bottleneck, effectively stripping away 2D spatial layout and structural identity clues, leaving only pure motion semantics.
Cross-Attention Control
Unlike standard approaches that add spatial offsets to the diffusion UNet, X-NeMo injects the 1D motion vector globally via cross-attention layers. This forces the network to semantically map the motion onto the reference identity rather than simply copying a spatial layout.
Dual-Head Latent Supervision
To capture high-frequency details like fine wrinkles or sharp expressions, the authors co-train a lightweight GAN decoder alongside the diffusion model. This GAN head uses perceptual and adversarial losses to ensure the 1D latent space captures fine-grained facial dynamics.
Training Strategies
- Reference Feature Masking: To prevent the model from ignoring the motion encoder when the reference and driving images are similar, 30% of reference appearance features are randomly masked during training.
- Heavy Augmentations: Aggressive spatial and color augmentations are applied to driving images to force the encoder to focus solely on motion rather than appearance.
Experimental Evaluation
The model was trained on large-scale datasets including HDTF, VFHQ, and the highly expressive NerSemble. Evaluation was conducted on a custom benchmark of 100 diverse in-the-wild portraits and 300 expressive test videos.
Key Results
X-NeMo significantly outperforms state-of-the-art baselines like LivePortrait and X-Portrait in both identity preservation and expression accuracy.
- Identity Preservation: X-NeMo achieved an Identity Similarity (ID-SIM) of 0.787, compared to the next best baseline at 0.725.
- Expression Accuracy: The model reached an Emotion Similarity (EMO-SIM) of 0.65, vastly outperforming X-Portrait (0.52).
- Qualitative Performance: Visually, X-NeMo handles extreme expressions—such as puffing cheeks or tongue protrusion—without distorting the reference face shape.
Conclusion and Future Work
X-NeMo demonstrates that global 1D latent descriptors are a powerful alternative to 2D spatial maps for cross-identity animation. By decoupling motion from identity, the model achieves a new state-of-the-art in expressive portrait reenactment. Future work involves integrating this motion control scheme into broader video diffusion backbones to enhance temporal consistency and explore applications like portrait video outpainting.
Summary updated