Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer

Jiahao Cui, Hui Li, Yun Zhan, Hanlin Shang, Kaihui Cheng, Yuqi Ma, Shan Mu, Hang Zhou, Jingdong Wang, Siyu Zhu

TL;DR

Hallo3 is the first framework to successfully adapt a pretrained Diffusion Transformer (DiT) video backbone for portrait image animation. By introducing a novel identity reference network and cross-attention audio injection, it generates highly dynamic, temporally consistent, long-duration talking head videos that seamlessly handle non-frontal poses, complex object interactions, and moving backgrounds.

What Problem Does This Paper Address?

Traditional portrait animation methods (such as SadTalker, LivePortrait, or EMO) rely heavily on U-Net architectures paired with explicit facial landmarks or 3D Morphable Models (3DMMs). While these methods excel at animating frontal, centered faces against static backgrounds, they struggle significantly with real-world complexities:

  • Extreme Poses: Side profiles, overhead, or low-angle views often cause severe facial distortions.
  • Dynamic Backgrounds: Traditional models assume static backdrops, failing to render realistic environments like crowded streets or moving scenery.
  • Foreground Interactions: Animating a subject interacting with accessories—like holding a smartphone or a microphone—remains an open challenge.

While Diffusion Transformers (DiTs) have revolutionized general video generation, their architecture renders previous U-Net-based conditioning mechanisms for identity preservation and audio synchronization obsolete. Hallo3 bridges this gap by designing new conditioning pipelines tailored specifically for DiT architectures.

Methods

The architecture of Hallo3 is built upon the CogVideoX baseline, a DiT model that uses a 3D Causal Variational Autoencoder (VAE) for video compression and a T5 text encoder. To make this backbone controllable via audio and reference images, the authors introduced three key components:

  1. Identity Reference Network: To prevent the subject's identity from drifting over long sequences, the authors designed a parallel reference network consisting of 42 transformer layers that share weights with the main denoising network. Vision tokens from the reference image are extracted and injected into the denoising network via self-attention layers. This is paired with a face encoder (InsightFace) to maintain fine-grained facial expressions via cross-attention.
  2. Audio-Driven Transformer Diffusion: Speech features are extracted using a pretrained Wav2Vec2 model and mapped to frame-specific representations through successive linear layers. Instead of using adaptive layer normalization (adaLN)—which struggles with complex sequential semantics—Hallo3 injects these audio embeddings using a dedicated cross-attention mechanism placed immediately after the face-attention layers.
  3. Temporal Motion Frames: To generate continuous, long-duration videos beyond the standard 49-frame limit, the model uses the final $n$ frames of the previously generated clip as a condition (padded with Gaussian noise) for the subsequent clip, ensuring seamless temporal transitions.

An overview of this architecture can be seen below:

Figure 2. The overview of the proposed method. Specifically, the method takes a refer- ence image, an audio sequence, and a textual prompt as inputs to generate a video out- put with temporal consistency and high visual fidelity. We leverage the casual 3D VAE, T5, and Wav2Vec models to process the visual, textual, and audio features, respectively. The Identity Reference Network extracts identity features from the input reference image and textual prompt, enabling controllable animation while preserving the subject’s appear- ance. The audio encoder generates motion information for lip synchronization, while the face encoder extracts facial features to maintain consistency in facial expressions. The 3D Full Attention and Audio-Attention Modules combine identity and motion data within a denoising network, producing high-fidelity, temporally consistent, and controllable ani- mated videos.

Datasets

The model was trained on a massive, curated dataset totaling 134 hours of video, filtered through a strict pipeline to isolate single speakers and ensure high-quality head/camera motion:

  • HDTF: 6 hours of high-quality talking head videos.
  • YouTube Data: 72 hours of public raw videos.
  • Movie Dataset: 56 hours of diverse film clips.

For evaluation, the authors utilized standard benchmarks (HDTF and Celeb-V) alongside a newly introduced Wild Dataset. This benchmark contains 34 representative real-world cases featuring extreme profile views, complex headwear, clothing, and highly dynamic foreground/background elements.

Key Insights and Ideas

  • The DiT Advantage for Backgrounds: Shifting to a DiT backbone allows the model to naturally learn environmental physics. Instead of just warping a face, the model can realistically animate a campfire flickering in front of a child or snow falling around a subject.
  • Cross-Attention Over adaLN for Audio: The authors discovered that while adaptive layer normalization works well for global conditions like text prompts, cross-attention is absolutely vital for sequential, highly semantic inputs like speech audio to achieve precise lip synchronization.
  • Dual Identity Conditioning: Relying solely on face embeddings tends to introduce blur and disrupt global visual context. Combining a structural Identity Reference Network (via self-attention) with a Face Encoder (via cross-attention) provides the optimal balance between preserving global identity and allowing local expression flexibility.
Figure 3. Different strategies of audio conditioning.

Key Results

  • State-of-the-Art Video Quality: Hallo3 achieved the lowest Frechet Inception Distance (FID) and Frechet Video Distance (FVD) on both standard benchmarks. On HDTF, it achieved an FVD of 160.838 (compared to Hallo's 173.497 and SadTalker's 203.860).
  • Unmatched Realism in the Wild: On the newly proposed Wild Dataset, Hallo3 significantly outperformed existing methods in motion expressiveness. It achieved a Subject Dynamic score of 13.286 (outperforming Hallo's 5.268) and a Background Dynamic score of 4.481 (outperforming Hallo's 1.272), while maintaining highly accurate lip-syncing metrics.

Limitations and Future Work

  • Lip-Sync Trade-off: Because the training data heavily emphasizes highly dynamic real-world videos with significant head and body movements, Hallo3 shows a slight metric disparity in pure lip-sync tightness compared to specialized, frontal-only SOTA models.
  • Computational Demand: Training a full DiT architecture with a parallel reference network is computationally heavy; the model required 64 NVIDIA A100 GPUs for training.

Future work will likely focus on optimizing inference efficiency and further tightening fine-grained lip synchronization during periods of intense body movement.

Summary updated

Abstract

Existing methodologies for animating portrait images face significant challenges, particularly in handling non-frontal perspectives, rendering dynamic objects around the portrait, and generating immersive, realistic backgrounds. In this paper, we introduce the first application of a pretrained transformer-based video generative model that demonstrates strong generalization capabilities and generates highly dynamic, realistic videos for portrait animation, effectively addressing these challenges. The adoption of a new video backbone model makes previous U-Net-based methods for identity maintenance, audio conditioning, and video extrapolation inapplicable. To address this limitation, we design an identity reference network consisting of a causal 3D VAE combined with a stacked series of transformer layers, ensuring consistent facial identity across video sequences. Additionally, we investigate various speech audio conditioning and motion frame mechanisms to enable the generation of continuous video driven by speech audio. Our method is validated through experiments on benchmark and newly proposed wild datasets, demonstrating substantial improvements over prior methods in generating realistic portraits characterized by diverse orientations within dynamic and immersive scenes. Further visualizations and the source code are available at: https://fudan-generative-vision.github.io/hallo3/.
BibTeX
@inproceedings{cui2025hallo,
  author = {Jiahao Cui and Hui Li and Yun Zhan and Hanlin Shang and Kaihui Cheng and Yuqi Ma and Shan Mu and Hang Zhou and Jingdong Wang and Siyu Zhu},
  title = {{Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=GBM75ZQDMV},
}
Figure