Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Diffusion Transformer Networks

Jiahao Cui, Hui Li, Yun Zhan, Hanlin Shang, Kaihui Cheng, Yuqi Ma, Shan Mu, Hang Zhou, Jingdong Wang, Siyu Zhu

Portrait image animation—the task of generating a talking video from a single static image—has seen significant progress. However, existing models often struggle with "wild" scenarios, such as non-frontal head poses, dynamic backgrounds, and complex interactions with objects. Hallo3 addresses these limitations by transitioning from traditional U-Net architectures to a powerful Diffusion Transformer (DiT) backbone.

Overview of the Hallo3 framework, demonstrating its ability to generate dynamic video sequences from a reference image, audio, and textual prompts.

Addressing the "Frontal Bias"

Most current portrait animation tools, such as SadTalker or EMO, rely on U-Net architectures. While effective for lip-syncing in controlled environments, these models exhibit a strong "frontal bias." They often fail when the subject is viewed from a profile or low angle, or when the background is not perfectly static. In contrast, Hallo3 leverages the superior generalization capabilities of a pretrained Video Diffusion Transformer (CogVideoX) to handle highly dynamic scenes and diverse perspectives.

Technical Methodology

The core of Hallo3 is the adaptation of the CogVideoX backbone for the specific task of portrait animation. This transition necessitates new mechanisms for identity preservation and audio conditioning.

Identity Reference Network

To maintain facial consistency across frames, the authors introduced an Identity Reference Network. This network utilizes a 3D Causal VAE to extract latent features from the reference image. These features are then integrated into the main denoising network via self-attention layers, ensuring that the subject's identity remains stable even during significant head movements or changes in perspective.

The detailed architecture of Hallo3, highlighting the Identity Reference Network, the audio conditioning pipeline using Wav2Vec, and the integration of motion frames.

Audio-Driven Motion and Extrapolation

Audio features are extracted using wav2vec 2.0. After investigating various conditioning mechanisms, the researchers found that a cross-attention strategy—where audio features serve as keys and values for visual queries—provides the most precise control over lip-syncing and facial expressions. For long-duration video generation, Hallo3 employs a "motion-frame" strategy, using the final frames of a generated segment as the initial condition for the subsequent one, ensuring temporal continuity.

Experimental Evaluation

The model was trained on a combination of standard benchmarks (HDTF, Celeb-V) and a newly curated "Wild" dataset. This wild dataset consists of 134 hours of high-quality video from YouTube and movies, specifically filtered to include diverse head poses and complex backgrounds.

Hallo3 achieves state-of-the-art results on several key metrics:

  • Visual Quality: It achieved the lowest FID (Fréchet Inception Distance) and FVD (Video FID) scores on benchmark datasets, indicating superior image and temporal quality.
  • Dynamic Realism: Significant improvements were noted in "Subject Dynamics" and "Background Dynamics" on the wild dataset compared to previous SOTA models like LivePortrait.
  • Qualitative Performance: The model successfully animates subjects in challenging scenarios, such as wearing accessories (hats, glasses) or interacting with objects (phones), where previous methods often produced visible artifacts.

Conclusion and Future Directions

Hallo3 represents a significant step forward in making portrait animation more realistic and versatile. By adopting a Transformer-based backbone, it overcomes the limitations of previous U-Net models in handling complex, dynamic environments. While there is a slight trade-off in lip-sync precision compared to some specialized models, the overall realism and generalization to "wild" conditions are vastly improved. Future work will focus on further refining lip-sync accuracy while maintaining the model's robust environmental understanding.

Summary updated

Abstract

Existing methodologies for animating portrait images face significant challenges, particularly in handling non-frontal perspectives, rendering dynamic objects around the portrait, and generating immersive, realistic backgrounds. In this paper, we introduce the first application of a pretrained transformer-based video generative model that demonstrates strong generalization capabilities and generates highly dynamic, realistic videos for portrait animation, effectively addressing these challenges. The adoption of a new video backbone model makes previous U-Net-based methods for identity maintenance, audio conditioning, and video extrapolation inapplicable. To address this limitation, we design an identity reference network consisting of a causal 3D VAE combined with a stacked series of transformer layers, ensuring consistent facial identity across video sequences. Additionally, we investigate various speech audio conditioning and motion frame mechanisms to enable the generation of continuous video driven by speech audio. Our method is validated through experiments on benchmark and newly proposed wild datasets, demonstrating substantial improvements over prior methods in generating realistic portraits characterized by diverse orientations within dynamic and immersive scenes. Further visualizations and the source code are available at: https://fudan-generative-vision.github.io/hallo3/.
BibTeX
@inproceedings{cui2025hallo,
  author = {Jiahao Cui and Hui Li and Yun Zhan and Hanlin Shang and Kaihui Cheng and Yuqi Ma and Shan Mu and Hang Zhou and Jingdong Wang and Siyu Zhu},
  title = {{Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Diffusion Transformer Networks}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=GBM75ZQDMV},
}
Figure