Motion-Conditioned Image Animation for Video Editing

Wilson Yan, Andrew Brown, Pieter Abbeel, Rohit Girdhar, Samaneh AzadiarXiv 2023 · DOI · arXiv

Abstract

Video editing requires a delicate balance between maintaining temporal consistency and adhering to semantic text prompts. Traditional methods often rely on rigid structural constraints like depth or edge maps, which preserve style but struggle with significant motion changes. We present MoCA (Motion-Conditioned Image Animation), a framework that treats video editing as a guided animation task. By decomposing the process into first-frame editing followed by motion-conditioned generation, MoCA enables both precise spatial edits and complex motion transformations. Central to our approach is a "togglable" optical flow constraint that allows the model to either preserve original movement or generate entirely new actions. Our results demonstrate a new state-of-the-art in video editing, particularly in challenging motion-based scenarios where existing models frequently fail.

Introduction

While image editing has seen rapid advancement, video editing remains a significant challenge. The primary difficulty lies in ensuring "flicker-free" temporal consistency while allowing for semantic flexibility. Current state-of-the-art (SOTA) models typically use dense structural guidance to stabilize frames. However, these rigid priors act as a double-edged sword: while they effectively maintain the layout for style transfers, they prevent the model from altering the underlying action. For instance, transforming a video of a sitting animal into one of a jumping animal is nearly impossible when the model is forced to adhere to the original sitting depth map.

MoCA addresses this by rethinking video editing as a decomposition problem: first, edit a single representative frame, and then animate that frame into a full sequence using the original video's motion as an optional, rather than mandatory, guide.

Methodology: The MoCA Framework

MoCA operates through a two-stage pipeline designed to leverage the strengths of existing 2D image editors and 3D temporal diffusion models.

1. First-Frame Editing

Given a source video, we first extract the initial frame. This frame is edited using off-the-shelf tools such as Prompt-to-Prompt or SDEdit based on the target text prompt. This stage ensures that the visual identity and style of the edit are established with high fidelity before any animation occurs.

2. Motion-Conditioned Animation

The core of MoCA is a Latent Diffusion Model (LDM) that extrapolates the edited first frame into a complete video. As shown in the architecture below, the model is conditioned on three primary inputs: the edit prompt, the edited first frame, and optical flow extracted from the source video.

The MoCA architecture decomposes editing into first-frame manipulation and motion-conditioned animation using a togglable flow constraint.

The "Flow Toggle" Innovation

The critical innovation in MoCA is the treatment of optical flow. For spatial edits (e.g., changing a background or style), the optical flow from the source video is used as a condition to ensure the new content follows the original motion. For motion edits (e.g., changing the action), the flow conditioning is "dropped out" during the inference phase. This allows the model to ignore the original movement and generate new temporal dynamics based solely on the text prompt, effectively switching from a "video-to-video" task to a "guided-image-to-video" task.

Evaluation and Results

To rigorously test MoCA, we introduced a benchmark of 271 tasks, including 117 new tasks specifically focused on motion-based edits (e.g., objects falling or changing direction). We compared MoCA against several leading baselines, including Dreamix, Gen-1, and MasaCtrl.

Qualitative comparison showing MoCA's ability to handle both spatial edits (sheep, waves) and complex motion edits (juggling, jumping) compared to baselines.

Performance Metrics

MoCA established a new SOTA across multiple dimensions:

  • Human Preference: In blind side-by-side evaluations, humans preferred MoCA over Dreamix (63%), MasaCtrl (75%), and Tune-A-Video (72%).
  • Motion Accuracy: The performance gap was most pronounced in motion-heavy edits. In the "Multi-Motion" category, MoCA achieved a 90% win rate against Gen-1, demonstrating its unique ability to break free from source video structures when prompted.
  • Quantitative Scores: Using the $M_{geo}$ metric—a geometric mean of CLIP directional and similarity scores—MoCA consistently outperformed all baselines in style, object, and background categories.

Discussion and Limitations

While MoCA excels at motion synthesis, it faces challenges with long-range extrapolation. Because the model relies heavily on the first frame, it can struggle with scenes involving extreme camera movement where new background elements must be "out-painted." Additionally, some fine details from the original source video may be lost if they were not present in the first frame, as the model essentially re-imagines the sequence from the starting point.

Conclusion

MoCA demonstrates that video editing does not always require rigid structural priors. By treating motion as a togglable condition and leveraging the power of first-frame image editing, MoCA provides a flexible and powerful framework for both stylistic and action-oriented video transformations. This approach paves the way for more creative and semantically rich video manipulation tools.

Summary updated

Abstract

We introduce MoCA, a Motion-Conditioned Image Animation approach for video editing. It leverages a simple decomposition of the video editing problem into image editing followed by motion-conditioned image animation. Furthermore, given the lack of robust evaluation datasets for video editing, we introduce a new benchmark that measures edit capability across a wide variety of tasks, such as object replacement, background changes, style changes, and motion edits. We present a comprehensive human evaluation of the latest video editing methods along with MoCA, on our proposed benchmark. MoCA establishes a new state-of-the-art, demonstrating greater human preference win-rate, and outperforming notable recent approaches including Dreamix (63%), MasaCtrl (75%), and Tune-A-Video (72%), with especially significant improvements for motion edits.
BibTeX
@misc{yan2023motion,
  author = {Wilson Yan and Andrew Brown and Pieter Abbeel and Rohit Girdhar and Samaneh Azadi},
  title = {{Motion-Conditioned Image Animation for Video Editing}},
  year = {2023},
  eprint = {2311.18827},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.18827},
}
Figure