Introduction
Generating realistic portrait videos, or "talking heads," is a cornerstone of digital media, yet it remains a significant challenge. Traditional methods often struggle to balance visual fidelity with computational efficiency. The researchers introduce SeMo (Semantic Latent Motion), a framework that moves beyond rigid, human-defined priors like facial landmarks. By employing a self-supervised, compact 1D motion token, SeMo achieves state-of-the-art realism in audio-driven portrait videos, boasting an 81% win rate in user studies.
The Challenge of Explicit Priors
Most existing portrait generation techniques follow one of two paths. The first relies on explicit human priors, such as 3D Morphable Models (3DMM) or facial landmarks. While structured, these designs are inherently biased; they often fail to capture subtle, non-facial nuances like the swing of jewelry or the bounce of hair, leading to the "uncanny valley" effect.
The second path utilizes large-scale diffusion models. While these produce high-quality imagery, they are computationally intensive because they operate on dense spatial-temporal data. SeMo seeks a middle ground: a representation that is compact enough for rapid inference but expressive enough to capture fine-grained human movement without hand-crafted rules.
The SeMo Framework
The SeMo architecture is divided into three distinct stages: Abstraction, Reasoning, and Generation.
1. Abstraction: The Masked Motion Encoder
Rather than pre-defining facial features, SeMo uses a self-supervised approach. The system takes a video frame and applies an extreme random mask—hiding up to 90% of the content. An encoder is then forced to compress the remaining information into a single 1D token (a 512-dimensional vector). This bottleneck forces the model to learn high-level semantic motion rather than memorizing specific pixels. This "extreme masking" ensures that identity information does not leak into the motion token, making it truly prior-free.
2. Reasoning: The Motion Generator
A Diffusion Transformer (DiT) is trained to predict these 1D motion tokens based on input audio signals (processed via OpenAI’s Whisper) and previous motion states. Because the representation is reduced to a single token per frame, this reasoning step is exceptionally fast.
3. Generation: The Motion Decoder
Finally, a rectified flow-based decoder synthesizes the target video frames using a reference identity image and the predicted motion tokens. Rectified flow allows for high-quality generation with significantly fewer steps than traditional diffusion models.
Key Results and Insights
SeMo demonstrates superior performance across both quantitative and qualitative benchmarks:
- Quantitative Excellence: On the HDTF dataset, SeMo achieved a Fréchet Inception Distance (FID) of 21.43, outperforming other state-of-the-art models like LivePortrait (22.46).
- Capturing the "Un-programmable": By avoiding fixed landmarks, SeMo naturally learns to animate details that landmarks ignore, such as the jiggle of earrings or complex hair movement during head turns.
- Robustness: The model handles challenging scenarios, such as "Edge Views" or partial faces, where traditional landmark detectors often fail and cause generation glitches.
- Efficiency: User studies confirmed a strong preference for SeMo, with an 81% win rate in realism compared to competing models. Users specifically highlighted the lack of "rigid-body" artifacts common in landmark-based systems.
Conclusion and Future Work
SeMo represents a shift toward self-supervised semantic representations in video generation. While currently limited to 256x256 resolution, the framework provides a highly efficient trade-off between compression and visual quality. Future work will focus on scaling the model to higher resolutions and exploring its application in complex, multi-subject scenes.
Summary updated