Semantic Latent Motion for Portrait Video Generation

Qiyuan Zhang, Chenyu Wu, Wenzhang Sun, Huaize Liu, Donglin Di, Wei Chen, Changqing ZouarXiv 2025 · DOI · arXiv

Introduction

Generating realistic portrait videos, or "talking heads," is a cornerstone of digital media, yet it remains a significant challenge. Traditional methods often struggle to balance visual fidelity with computational efficiency. The researchers introduce SeMo (Semantic Latent Motion), a framework that moves beyond rigid, human-defined priors like facial landmarks. By employing a self-supervised, compact 1D motion token, SeMo achieves state-of-the-art realism in audio-driven portrait videos, boasting an 81% win rate in user studies.

The Challenge of Explicit Priors

Most existing portrait generation techniques follow one of two paths. The first relies on explicit human priors, such as 3D Morphable Models (3DMM) or facial landmarks. While structured, these designs are inherently biased; they often fail to capture subtle, non-facial nuances like the swing of jewelry or the bounce of hair, leading to the "uncanny valley" effect.

The second path utilizes large-scale diffusion models. While these produce high-quality imagery, they are computationally intensive because they operate on dense spatial-temporal data. SeMo seeks a middle ground: a representation that is compact enough for rapid inference but expressive enough to capture fine-grained human movement without hand-crafted rules.

Comparison of motion representations (top) and generation methods (bottom). SeMo introduces a prior-free implicit motion representation and motion-conditioned decoding.

The SeMo Framework

The SeMo architecture is divided into three distinct stages: Abstraction, Reasoning, and Generation.

1. Abstraction: The Masked Motion Encoder

Rather than pre-defining facial features, SeMo uses a self-supervised approach. The system takes a video frame and applies an extreme random mask—hiding up to 90% of the content. An encoder is then forced to compress the remaining information into a single 1D token (a 512-dimensional vector). This bottleneck forces the model to learn high-level semantic motion rather than memorizing specific pixels. This "extreme masking" ensures that identity information does not leak into the motion token, making it truly prior-free.

2. Reasoning: The Motion Generator

A Diffusion Transformer (DiT) is trained to predict these 1D motion tokens based on input audio signals (processed via OpenAI’s Whisper) and previous motion states. Because the representation is reduced to a single token per frame, this reasoning step is exceptionally fast.

3. Generation: The Motion Decoder

Finally, a rectified flow-based decoder synthesizes the target video frames using a reference identity image and the predicted motion tokens. Rectified flow allows for high-quality generation with significantly fewer steps than traditional diffusion models.

The three-stage SeMo architecture: Abstraction (Motion Encoder), Generation (Motion Decoder), and Reasoning (Motion Generator).

Key Results and Insights

SeMo demonstrates superior performance across both quantitative and qualitative benchmarks:

  • Quantitative Excellence: On the HDTF dataset, SeMo achieved a Fréchet Inception Distance (FID) of 21.43, outperforming other state-of-the-art models like LivePortrait (22.46).
  • Capturing the "Un-programmable": By avoiding fixed landmarks, SeMo naturally learns to animate details that landmarks ignore, such as the jiggle of earrings or complex hair movement during head turns.
  • Robustness: The model handles challenging scenarios, such as "Edge Views" or partial faces, where traditional landmark detectors often fail and cause generation glitches.
  • Efficiency: User studies confirmed a strong preference for SeMo, with an 81% win rate in realism compared to competing models. Users specifically highlighted the lack of "rigid-body" artifacts common in landmark-based systems.

Conclusion and Future Work

SeMo represents a shift toward self-supervised semantic representations in video generation. While currently limited to 256x256 resolution, the framework provides a highly efficient trade-off between compression and visual quality. Future work will focus on scaling the model to higher resolutions and exploring its application in complex, multi-subject scenes.

Summary updated

Abstract

Recent advancements in portrait video generation have been noteworthy. However, existing methods rely heavily on human priors and pre-trained generative models, Motion representations based on human priors may introduce unrealistic motion, while methods relying on pre-trained generative models often suffer from inefficient inference. To address these challenges, we propose Semantic Latent Motion (SeMo), a compact and expressive motion representation. Leveraging this representation, our approach achieve both high-quality visual results and efficient inference. SeMo follows an effective three-step framework: Abstraction, Reasoning, and Generation. First, in the Abstraction step, we use a carefully designed Masked Motion Encoder, which leverages a self-supervised learning paradigm to compress the subject's motion state into a compact and abstract latent motion (1D token). Second, in the Reasoning step, we efficiently generate motion sequences based on the driving audio signal. Finally, in the Generation step, the motion dynamics serve as conditional information to guide the motion decoder in synthesizing realistic transitions from reference frame to target video. Thanks to the compact and expressive nature of Semantic Latent Motion, our method achieves efficient motion representation and high-quality video generation. User studies demonstrate that our approach surpasses state-of-the-art models with an 81% win rate in realism. Extensive experiments further highlight its strong compression capability, reconstruction quality, and generative potential.
BibTeX
@misc{zhang2025semantic,
  author = {Qiyuan Zhang and Chenyu Wu and Wenzhang Sun and Huaize Liu and Donglin Di and Wei Chen and Changqing Zou},
  title = {{Semantic Latent Motion for Portrait Video Generation}},
  year = {2025},
  eprint = {2503.10096},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.10096},
}
Figure