CrossPortrait: Cross-Species Portrait Animation via Adversarial Motion Alignment

Gezhong Pan, Zhe Li, Yunfan Tao, Yongheng Zhao, Shiqi Zhou, Juyong Zhang, Xiaosong Jia, Yanyong Zhang, Shuquan Ye, Shaofeng ZhangarXiv 2026 · arXiv

TL;DR

CrossPortrait is a new framework for cross-species portrait animation that allows a driving video of one species (e.g., a human) to animate a source portrait of another (e.g., a cat or dog). By combining species-specific LoRA adapters with adversarial domain adaptation, the model successfully transfers complex expressions while preventing "species leakage"—the uncanny effect where the animated animal begins to take on the anatomical features of the driving human.

What Problem Does This Paper Address?

Portrait animation has seen massive strides recently, but most models are "species-locked." If you try to use a human video to animate a cat using standard methods, you often encounter two major hurdles:

  1. Anatomical Mismatch: Humans, cats, and dogs have vastly different facial structures. A human smile involves different muscle movements and geometry than a cat’s snarl or a dog’s pant.
  2. Species Leakage: This is the "uncanny valley" of cross-species work. When a human drives an animal portrait, the resulting animation often bleeds human traits into the animal—like human-shaped teeth appearing in a cat's mouth or the animal's snout morphing to match a human nose.
Comparison of species leakage: notice how baselines transfer human dental characteristics to the cat, whereas CrossPortrait preserves the cat's morphology.

Methods

The authors propose a two-stage framework that treats rigid head motion and non-rigid facial expressions separately.

Stage 1: Learning Motion Alignment

The core innovation lies in how expressions are encoded. The authors introduce a species-routed expression encoder. This uses a shared backbone to find a "common language" for expressions across species, but injects species-specific LoRA (Low-Rank Adaptation) components. These adapters allow the model to handle the unique anatomical nuances of each species without losing the shared representation.

To fight species leakage, they employ Expression-level Adversarial Domain Adaptation (Ex-ADA). They train a domain classifier to identify the species from the expression features. Simultaneously, they use a Gradient Reversal Layer (GRL) to force the expression encoder to suppress species-identifying information. This ensures the latent features represent "the expression" (e.g., mouth open) rather than "a human mouth opening."

Stage 1 architecture showing the species-routed encoder and the adversarial domain adaptation (Ex-ADA) loop.

Stage 2: Video Generation

The frozen encoders from Stage 1 provide motion guidance to a Diffusion Transformer (DiT) backbone. Rigid pose is injected as a dense pose-flow field, while expressions are converted into tokens and injected via cross-attention. A lightweight causal temporal module is added to ensure the resulting video is smooth and consistent over time.

Datasets

The model was trained on a diverse mix of human and animal data:

  • Human Data: Public datasets including NeRSemble, AVA-256, PFHQ, and VFHQ.
  • Animal Data: Internally collected high-quality videos of cats and dogs, totaling approximately 15 hours of footage per animal domain.
  • Evaluation: A dedicated animal self-reenactment benchmark consisting of 50 cat and 50 dog videos was used to measure reconstruction accuracy.

Key Insights and Ideas

  • Disentanglement is Essential: By separating rigid pose (which is mathematically similar across species) from expression (which is anatomically distinct), the model avoids complex geometric warping issues.
  • Adversarial "Forgetting": The use of Ex-ADA is a clever way to ensure the model learns a truly species-agnostic expression space. By actively trying to "hide" the species from the classifier, the encoder focuses purely on the semantics of the motion.
  • LoRAs for Anatomy: Instead of training entirely separate models for each species, LoRAs provide just enough "local" knowledge to handle a cat's whiskers or a dog's ears while keeping the overall framework unified.

Key Results

CrossPortrait significantly outperforms state-of-the-art baselines like LivePortrait and Wan-Animate-2 in cross-species tasks.

  • Quantitative: In animal self-reenactment, it achieved the highest PSNR (25.06) and lowest LPIPS (0.1360), indicating superior reconstruction quality. In human-to-animal (H→A) user studies, it was preferred over the No-Alignment baseline by a wide margin (only 14.1% overall preference for the baseline).
  • Qualitative: The model maintains the source animal's identity and morphology (e.g., keeping the correct shape of a cat's tongue or teeth) even when driven by an expressive human face.
Qualitative results showing successful animation across human-to-animal, animal-to-human, and cross-species animal scenarios.

Limitations and Future Work

  • Limited Species Palette: While the model generalizes well to unseen identities of cats and dogs, it is currently trained only on humans, cats, and dogs. Generalizing to entirely different topologies (like birds or horses) remains a challenge.
  • Driving Constraints: The model currently requires a driving video from one of the three trained species. Using an "unseen" species as a driver is not yet supported.
  • Metric Gap: The authors note a lack of standardized, biologically-grounded metrics for cross-species animation, relying heavily on user studies and identity-preservation scores (CSIM). Developing more robust automated metrics for this sub-field is a clear next step.

Summary updated

Abstract

Cross-species portrait animation, which aims to animate a source portrait of species A using a driving video of species B, is particularly challenging due to substantial anatomical differences in facial structures across species. Previous works typically fine-tune human portrait animation models on animal video datasets but often suffer from driving-species leakage. To address this, we propose CrossPortrait, a novel framework for expressive cross-species portrait animation, supporting diverse scenarios including human-to-animal, animal-to-human and cross-species animal-to-animal (e.g. cat-to-dog) animation. Specifically, we decompose the portrait motion into rigid head poses and expression keypoints. To accommodate species-specific facial structures while facilitating cross-species expression alignment, we introduce a species-routed expression encoder that incorporates species-specific LoRAs into a unified expression encoder. Furthermore, we introduce expression-level adversarial domain adaptation (Ex-ADA) to suppress driving-species-specific information in the expression latents, thereby mitigating species leakage and promoting a shared expression space across species. Extensive experiments show that our method enables expressive and identity-preserving portrait animation across diverse species and outperforms other state-of-the-art approaches.
BibTeX
@misc{pan2026crossportrait,
  author = {Gezhong Pan and Zhe Li and Yunfan Tao and Yongheng Zhao and Shiqi Zhou and Juyong Zhang and Xiaosong Jia and Yanyong Zhang and Shuquan Ye and Shaofeng Zhang},
  title = {{CrossPortrait: Cross-Species Portrait Animation via Adversarial Motion Alignment}},
  year = {2026},
  eprint = {2610.17532},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.17532},
}
Figure