TL;DR
Hallo3 is the first framework to successfully adapt a pretrained Diffusion Transformer (DiT) video backbone for portrait image animation. By introducing a novel identity reference network and cross-attention audio injection, it generates highly dynamic, temporally consistent, long-duration talking head videos that seamlessly handle non-frontal poses, complex object interactions, and moving backgrounds.
What Problem Does This Paper Address?
Traditional portrait animation methods (such as SadTalker, LivePortrait, or EMO) rely heavily on U-Net architectures paired with explicit facial landmarks or 3D Morphable Models (3DMMs). While these methods excel at animating frontal, centered faces against static backgrounds, they struggle significantly with real-world complexities:
- Extreme Poses: Side profiles, overhead, or low-angle views often cause severe facial distortions.
- Dynamic Backgrounds: Traditional models assume static backdrops, failing to render realistic environments like crowded streets or moving scenery.
- Foreground Interactions: Animating a subject interacting with accessories—like holding a smartphone or a microphone—remains an open challenge.
While Diffusion Transformers (DiTs) have revolutionized general video generation, their architecture renders previous U-Net-based conditioning mechanisms for identity preservation and audio synchronization obsolete. Hallo3 bridges this gap by designing new conditioning pipelines tailored specifically for DiT architectures.
Methods
The architecture of Hallo3 is built upon the CogVideoX baseline, a DiT model that uses a 3D Causal Variational Autoencoder (VAE) for video compression and a T5 text encoder. To make this backbone controllable via audio and reference images, the authors introduced three key components:
- Identity Reference Network: To prevent the subject's identity from drifting over long sequences, the authors designed a parallel reference network consisting of 42 transformer layers that share weights with the main denoising network. Vision tokens from the reference image are extracted and injected into the denoising network via self-attention layers. This is paired with a face encoder (InsightFace) to maintain fine-grained facial expressions via cross-attention.
- Audio-Driven Transformer Diffusion: Speech features are extracted using a pretrained Wav2Vec2 model and mapped to frame-specific representations through successive linear layers. Instead of using adaptive layer normalization (adaLN)—which struggles with complex sequential semantics—Hallo3 injects these audio embeddings using a dedicated cross-attention mechanism placed immediately after the face-attention layers.
- Temporal Motion Frames: To generate continuous, long-duration videos beyond the standard 49-frame limit, the model uses the final $n$ frames of the previously generated clip as a condition (padded with Gaussian noise) for the subsequent clip, ensuring seamless temporal transitions.
An overview of this architecture can be seen below:
Datasets
The model was trained on a massive, curated dataset totaling 134 hours of video, filtered through a strict pipeline to isolate single speakers and ensure high-quality head/camera motion:
- HDTF: 6 hours of high-quality talking head videos.
- YouTube Data: 72 hours of public raw videos.
- Movie Dataset: 56 hours of diverse film clips.
For evaluation, the authors utilized standard benchmarks (HDTF and Celeb-V) alongside a newly introduced Wild Dataset. This benchmark contains 34 representative real-world cases featuring extreme profile views, complex headwear, clothing, and highly dynamic foreground/background elements.
Key Insights and Ideas
- The DiT Advantage for Backgrounds: Shifting to a DiT backbone allows the model to naturally learn environmental physics. Instead of just warping a face, the model can realistically animate a campfire flickering in front of a child or snow falling around a subject.
- Cross-Attention Over adaLN for Audio: The authors discovered that while adaptive layer normalization works well for global conditions like text prompts, cross-attention is absolutely vital for sequential, highly semantic inputs like speech audio to achieve precise lip synchronization.
- Dual Identity Conditioning: Relying solely on face embeddings tends to introduce blur and disrupt global visual context. Combining a structural Identity Reference Network (via self-attention) with a Face Encoder (via cross-attention) provides the optimal balance between preserving global identity and allowing local expression flexibility.
Key Results
- State-of-the-Art Video Quality: Hallo3 achieved the lowest Frechet Inception Distance (FID) and Frechet Video Distance (FVD) on both standard benchmarks. On HDTF, it achieved an FVD of 160.838 (compared to Hallo's 173.497 and SadTalker's 203.860).
- Unmatched Realism in the Wild: On the newly proposed Wild Dataset, Hallo3 significantly outperformed existing methods in motion expressiveness. It achieved a Subject Dynamic score of 13.286 (outperforming Hallo's 5.268) and a Background Dynamic score of 4.481 (outperforming Hallo's 1.272), while maintaining highly accurate lip-syncing metrics.
Limitations and Future Work
- Lip-Sync Trade-off: Because the training data heavily emphasizes highly dynamic real-world videos with significant head and body movements, Hallo3 shows a slight metric disparity in pure lip-sync tightness compared to specialized, frontal-only SOTA models.
- Computational Demand: Training a full DiT architecture with a parallel reference network is computationally heavy; the model required 64 NVIDIA A100 GPUs for training.
Future work will likely focus on optimizing inference efficiency and further tightening fine-grained lip synchronization during periods of intense body movement.
Summary updated