Abstract
Video editing requires a delicate balance between maintaining temporal consistency and adhering to semantic text prompts. Traditional methods often rely on rigid structural constraints like depth or edge maps, which preserve style but struggle with significant motion changes. We present MoCA (Motion-Conditioned Image Animation), a framework that treats video editing as a guided animation task. By decomposing the process into first-frame editing followed by motion-conditioned generation, MoCA enables both precise spatial edits and complex motion transformations. Central to our approach is a "togglable" optical flow constraint that allows the model to either preserve original movement or generate entirely new actions. Our results demonstrate a new state-of-the-art in video editing, particularly in challenging motion-based scenarios where existing models frequently fail.
Introduction
While image editing has seen rapid advancement, video editing remains a significant challenge. The primary difficulty lies in ensuring "flicker-free" temporal consistency while allowing for semantic flexibility. Current state-of-the-art (SOTA) models typically use dense structural guidance to stabilize frames. However, these rigid priors act as a double-edged sword: while they effectively maintain the layout for style transfers, they prevent the model from altering the underlying action. For instance, transforming a video of a sitting animal into one of a jumping animal is nearly impossible when the model is forced to adhere to the original sitting depth map.
MoCA addresses this by rethinking video editing as a decomposition problem: first, edit a single representative frame, and then animate that frame into a full sequence using the original video's motion as an optional, rather than mandatory, guide.
Methodology: The MoCA Framework
MoCA operates through a two-stage pipeline designed to leverage the strengths of existing 2D image editors and 3D temporal diffusion models.
1. First-Frame Editing
Given a source video, we first extract the initial frame. This frame is edited using off-the-shelf tools such as Prompt-to-Prompt or SDEdit based on the target text prompt. This stage ensures that the visual identity and style of the edit are established with high fidelity before any animation occurs.
2. Motion-Conditioned Animation
The core of MoCA is a Latent Diffusion Model (LDM) that extrapolates the edited first frame into a complete video. As shown in the architecture below, the model is conditioned on three primary inputs: the edit prompt, the edited first frame, and optical flow extracted from the source video.
The "Flow Toggle" Innovation
The critical innovation in MoCA is the treatment of optical flow. For spatial edits (e.g., changing a background or style), the optical flow from the source video is used as a condition to ensure the new content follows the original motion. For motion edits (e.g., changing the action), the flow conditioning is "dropped out" during the inference phase. This allows the model to ignore the original movement and generate new temporal dynamics based solely on the text prompt, effectively switching from a "video-to-video" task to a "guided-image-to-video" task.
Evaluation and Results
To rigorously test MoCA, we introduced a benchmark of 271 tasks, including 117 new tasks specifically focused on motion-based edits (e.g., objects falling or changing direction). We compared MoCA against several leading baselines, including Dreamix, Gen-1, and MasaCtrl.
Performance Metrics
MoCA established a new SOTA across multiple dimensions:
- Human Preference: In blind side-by-side evaluations, humans preferred MoCA over Dreamix (63%), MasaCtrl (75%), and Tune-A-Video (72%).
- Motion Accuracy: The performance gap was most pronounced in motion-heavy edits. In the "Multi-Motion" category, MoCA achieved a 90% win rate against Gen-1, demonstrating its unique ability to break free from source video structures when prompted.
- Quantitative Scores: Using the $M_{geo}$ metric—a geometric mean of CLIP directional and similarity scores—MoCA consistently outperformed all baselines in style, object, and background categories.
Discussion and Limitations
While MoCA excels at motion synthesis, it faces challenges with long-range extrapolation. Because the model relies heavily on the first frame, it can struggle with scenes involving extreme camera movement where new background elements must be "out-painted." Additionally, some fine details from the original source video may be lost if they were not present in the first frame, as the model essentially re-imagines the sequence from the starting point.
Conclusion
MoCA demonstrates that video editing does not always require rigid structural priors. By treating motion as a togglable condition and leveraging the power of first-frame image editing, MoCA provides a flexible and powerful framework for both stylistic and action-oriented video transformations. This approach paves the way for more creative and semantically rich video manipulation tools.
Summary updated