LivePhoto: Real Image Animation with Text-guided Motion Control

Xi Chen, Zhiheng Liu, Mengting Chen, Yutong Feng, Yu Liu, Yujun Shen, Hengshuang Zhao

Abstract

Animating a single still image into a coherent video remains a significant challenge in computer vision, primarily due to motion ambiguity and the conflict between image content and text-based motion instructions. We present LivePhoto, a novel framework that empowers users with precise, text-guided motion control. By decoupling content from motion and introducing a numerical "motion intensity" slider, LivePhoto ensures high identity preservation while faithfully executing complex motion prompts.

Introduction: The Challenge of Creative Control

Current text-to-video models often exhibit a "creative mind of their own," frequently ignoring specific motion instructions or altering the identity of the subject in the reference image. For instance, a prompt like "a dog running fast" might result in a slow walk or a change in the dog's breed. The authors identify two primary hurdles:

  1. Motion Ambiguity: Textual descriptions are often too vague to specify the exact magnitude of motion. "Shaking head" could mean a subtle tilt or a violent thrash.
  2. Content-Motion Conflict: Prompts often describe both the scene content and the motion. Models may prioritize the "what" over the "how," leading them to redraw the subject rather than animating the existing one.

Methodology: Steering the Animation

LivePhoto is built upon Stable Diffusion (v1.5), enhanced with three core steering mechanisms to ensure fidelity and control.

Image Content Guidance

To maintain the integrity of the source image, LivePhoto employs a multi-layered approach:

  • Latent Feature Injection: Original image features are fed directly into the model's latent space.
  • DINOv2 Encoder: Captures global identity and structural details to ensure the subject remains recognizable.
  • Prior Inversion: The animation process begins with a noisy version of the reference image instead of random static, providing a strong structural anchor.

Motion Intensity Estimation

A key innovation is the parameterization of motion. The researchers discovered that the Structural Similarity Index (SSIM)—a metric usually used for image quality—serves as an excellent proxy for perceived motion. By calculating SSIM between frames in training data, they established a 1-to-10 intensity scale. This allows users to adjust the "energy" of the animation via a simple slider during inference.

Text Re-weighting Module

To resolve content-motion conflicts, a specialized module identifies motion-specific tokens (e.g., "running," "zooming," "wagging") and boosts their influence within the cross-attention layers. This prevents the model from being distracted by descriptive content that should be derived from the reference image.

Key Results and Insights

LivePhoto was evaluated against industry leaders like Gen-2 and Pikalabs using the WebVID dataset and a custom multi-category benchmark covering humans, animals, cartoons, and scenery.

LivePhoto demonstrates precise motion control across various categories, including intensity adjustment.

Performance Metrics

  • Text Consistency: In user studies, LivePhoto achieved a score of 4.7/5.0, significantly higher than competitors who hovered between 2.5 and 3.5.
  • Identity Preservation: The combination of DINOv2 and Prior Inversion resulted in superior subject consistency compared to open-source alternatives like AnimateDiff-I2V.
  • Quantitative Superiority: The model reached a DINO Score of 90.8 and a CLIP Score of 95.2, indicating excellent frame-to-frame stability and prompt alignment.

Creative Capabilities

Beyond simple movement, LivePhoto can "conjure" dynamic elements not present in the original frame, such as pouring liquid into an empty glass or generating lightning in a clear night sky, while maintaining the background's static integrity.

Conclusion and Future Directions

While LivePhoto sets a new standard for controllable image animation, it currently operates at a 256x256 resolution. Future work involves porting the framework to higher-capacity models like SD-XL to enhance visual fidelity. Additionally, the researchers aim to further refine the trade-off between high-intensity motion and potential motion blur to ensure clarity even at maximum energy levels.

Summary updated

BibTeX
@inproceedings{chen2024livephoto,
  author = {Xi Chen and Zhiheng Liu and Mengting Chen and Yutong Feng and Yu Liu and Yujun Shen and Hengshuang Zhao},
  title = {{LivePhoto: Real Image Animation with Text-guided Motion Control}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-72649-1\_27},
}
Figure