Edit-Your-Motion: Space-Time Diffusion Decoupling Learning for Video Motion Editing

Yi Zuo, Lingling Li, Licheng Jiao, Fang Liu, Xu Liu, Wenping Ma, Shuyuan Yang, Yuwei Guo

TL;DR

This paper introduces "Edit-Your-Motion," a novel one-shot video motion editing technique that effectively disentangles human motion from appearance. By leveraging DDIM inversion for appearance preservation and a new spatio-temporal decoupling strategy, it achieves high-fidelity motion editing on unseen "in-the-wild" videos, overcoming common issues like ghosting and distortion.

What Problem Does This Paper Address?

Existing diffusion-based methods for video motion editing have shown impressive results, but they often struggle with "in-the-wild" scenarios – real-world videos with complex backgrounds and unpredictable lighting. A key challenge is maintaining the original appearance and background of the source video while accurately transferring new motion from a reference skeleton. Many current approaches either fail to decouple motion and appearance effectively, leading to ghosting and body distortions, or require extensive training data and computational resources, making them impractical for diverse, unseen scenarios.

Methods

Edit-Your-Motion tackles these challenges with a multi-pronged approach:

  • DDIM Inversion for Appearance Preservation: Instead of starting from random noise, the method uses DDIM inversion to initialize the diffusion process with latent noise that already encodes the appearance of the source video. This ensures that the edited video retains the original visual characteristics.
  • Motion Attention Module (MA): A lightweight adapter module is introduced to enhance motion fidelity. It uses a combination of self-attention, cross-attention, and temporal attention to resolve conflicts between skeleton (motion) features and appearance features, leading to better alignment.
  • Spatio-Temporal Decoupling Two-Stage Learning (STL): This is a core innovation. The training process is split into two stages:
  • Stage 1 (Temporal Focus): The model learns temporal motion features by masking out the background and focusing solely on human appearance and motion.
  • Stage 2 (Spatial Focus): The model then learns spatial features from both the human appearance and the background, allowing for a more complete understanding of the scene.
  • Recurrent Causal Attention (RCA): To improve inter-frame consistency, standard spatial attention is replaced with RCA. RCA directly models dependencies between consecutive frames (both previous and next), ensuring smoother transitions and a more coherent video output.

Datasets

The paper uses the following datasets:

  • TikTok dataset: Contains 340 video sequences with corresponding masks and sensepose annotations.
  • YouTube dataset: A collection of 80 "in-the-wild" video sequences, each with at least 70 frames, used to evaluate the model's robustness on unseen data.

Key Insights and Ideas

  • The importance of decoupling: The paper strongly emphasizes that effectively separating spatial (appearance/background) and temporal (motion) features is crucial for robust video motion editing, especially in challenging "in-the-wild" scenarios.
  • One-shot fine-tuning is key: For practical applications, the ability to adapt to new, unseen videos with minimal training (one-shot fine-tuning) is a significant advantage over methods requiring extensive retraining.
  • DDIM inversion as a powerful initialization: Using DDIM inversion to preserve appearance information at the start of the diffusion process is a clever way to avoid ghosting and maintain visual fidelity.
  • RCA for enhanced temporal coherence: Explicitly modeling connections between past and future frames with RCA proves more effective than standard attention for maintaining smooth and consistent video sequences.

Key Results

  • Quantitative improvements: Edit-Your-Motion outperforms several state-of-the-art methods (including MagicAnimate, AnimateAnyone, Champ, MotionDirector, Follow-Your-Pose, Tune-A-Video, and MotionEditor) across various metrics like L1, PSNR, SSIM, LPIPS, FID-VID, and FVD on both the TikTok and "in-the-wild" datasets. For instance, on unseen "in-the-wild" cases, it achieves a higher PSNR (23.03) and SSIM (0.846) compared to other methods.
  • Qualitative superiority: Visualizations show that Edit-Your-Motion successfully aligns motion with reference skeletons while preserving the appearance and background of the source video, avoiding the distortions and ghosting seen in competing methods.
  • User study validation: A user study with 70 participants indicated that Edit-Your-Motion is preferred for text alignment, content alignment, and motion alignment compared to other methods.

Limitations and Future Work

  • Limitations: The paper acknowledges that while Edit-Your-Motion significantly improves upon existing methods, there might still be subtle inconsistencies or artifacts in highly complex scenarios. The effectiveness of the skeleton offset algorithm in mitigating ghosting is mentioned, implying that ghosting can still be a challenge.
  • Future Work: The authors suggest that Edit-Your-Motion can be applied to various multimedia fields like animations, advertisements, and short videos, enabling users to create more engaging content. They also hint at future work involving further improvements to robustness and potentially extending the method to more complex editing tasks.

Summary updated

Abstract

Existing diffusion-based video editing methods have achieved impressive results in motion editing. Most of the existing methods focus on the motion alignment between the edited video and the reference video. However, these methods do not constrain the background and object content of the video to remain unchanged, which makes it possible for users to generate unexpected videos. In this paper, we propose a one-shot video motion editing method called Edit-Your-Motion that requires only a single text-video pair for training. Specifically, we design the Detailed Prompt-Guided Learning Strategy (DPL) to decouple spatio-temporal features in space-time diffusion models. DPL separates learning object content and motion into two training stages. In the first training stage, we focus on learning the spatial features (the features of object content) and breaking down the temporal relationships in the video frames by shuffling them. We further propose Recurrent-Causal Attention (RC-Attn) to learn the consistent content features of the object from unordered video frames. In the second training stage, we restore the temporal relationship in video frames to learn the temporal feature (the features of the background and object's motion). We also adopt the Noise Constraint Loss to smooth out inter-frame differences. Finally, in the inference stage, we inject the content features of the source object into the editing branch through a two-branch structure (editing branch and reconstruction branch). With Edit-Your-Motion, users can edit the motion of objects in the source video to generate more exciting and diverse videos. Comprehensive qualitative experiments, quantitative experiments and user preference studies demonstrate that Edit-Your-Motion performs better than other methods.
BibTeX
@article{zuo2024edit,
  author = {Yi Zuo and Lingling Li and Licheng Jiao and Fang Liu and Xu Liu and Wenping Ma and Shuyuan Yang and Yuwei Guo},
  title = {{Edit-Your-Motion: Space-Time Diffusion Decoupling Learning for Video Motion Editing}},
  journal = {IEEE Transactions on Multimedia},
  year = {2024},
  doi = {10.1109/tmm.2026.3651137},
}
Figure