Motion-Aware Concept Alignment for Consistent Video Editing

Tong Zhang, Juan C Leon Alcazar, Victor Escorcia, Bernard GhanemSubmission to ICLR 2026 · Rejected

TL;DR

MoCA-Video is a training-free framework that enables "semantic mixing" in videos—the ability to blend features from a reference image into a video object (e.g., turning a cat into a "cat-astronaut"). By intelligently manipulating latent noise trajectories with momentum-based correction and robust object tracking, it produces high-quality, temporally consistent hybrid entities that transcend the original training distributions of diffusion models.

What Problem Does This Paper Address?

While video diffusion models have become incredibly powerful, they are still largely "stuck" within their training data. If you want to create a hybrid entity that the model hasn't seen before—like a corgi-shaped coffee machine or a feline astronaut—standard methods often fail.

Existing video editing techniques typically fall into two traps: they either apply global style transfers that wash out local details, or they perform frame-by-frame edits that lead to jarring "flicker" and temporal inconsistency. The challenge lies in performing fine-grained, region-specific semantic mixing while ensuring the object moves naturally and stays visually coherent across the entire video sequence.

Methods

MoCA-Video operates in the latent space of a frozen video diffusion model (VideoCrafter2). It avoids the need for expensive retraining by using three core components:

  1. Latent Space Tracking: To ensure the edit stays localized, the system uses Grounded-SAM2 to segment the target object. Because the latent space is noisy, the authors decode a "proxy" clean image ($x_0$) at specific timesteps to generate masks. They then use an IoU-based overlap maximization strategy to track these masks across frames, preventing "spatial drift."
  2. Momentum-Corrected DDIM: This is the technical heart of the paper. Standard denoising stays within the model's known data manifold. MoCA-Video introduces a momentum term that accumulates residual changes from semantic injection. This effectively "nudges" the denoising trajectory toward a novel hybrid distribution, allowing the model to navigate toward unseen semantic combinations (like the cat-astronaut).
  3. Gamma Residual Noise: To combat the flickering often caused by latent manipulation, the authors inject a small, calibrated amount of $\gamma$-scaled noise at each step. This acts as a lightweight regularizer, smoothing out visual artifacts and ensuring a stable transition between frames.

Datasets

The authors introduced a new benchmark to evaluate this specific task:

  • Entity Blending Dataset: Built upon the CTIB dataset and extended with 16 subcategories from the DAVIS-16 video segmentation dataset.
  • It consists of 100 unique entity pairs spanning intra-category (e.g., cow and sheep) and inter-category (e.g., astronaut and cat) blends.
  • Each entry includes a source prompt, a base video, a reference image, and a blending strength parameter ($\lambda$).

Key Insights and Ideas

  • Trajectory over Frames: The paper argues that high-quality video editing isn't about changing pixels frame-by-frame; it’s about structured manipulation of the noise trajectory. By sharing semantic information across frames via a diagonal denoising scheduler, the model maintains better temporal "memory."
  • Escaping the Manifold: The use of momentum is a clever way to bypass the limitations of a frozen model. It allows the framework to generate concepts that the base model was never explicitly trained to understand.
  • The CASS Metric: The authors recognized that standard metrics like SSIM (structural similarity) don't capture "blending quality." They proposed the Conceptual Alignment Shift Score (CASS), which uses CLIP embeddings to measure how much the video has shifted away from the original text prompt and toward the reference image.

Key Results

MoCA-Video consistently outperformed both training-free (FreeBlend, RAVE) and pre-trained (TokenFlow, AnimateDiffV2V) baselines:

  • Semantic Alignment: MoCA-Video achieved a CASS of 4.93, significantly higher than the next best training-free method, RAVE (3.80).
  • Temporal Stability: It maintained a low LPIPS-T (0.11), indicating smooth motion and minimal flickering compared to other high-edit methods.
  • Ablation Impact: The study found that IoU-based tracking was the most critical factor for spatial fidelity (removing it caused a 20% drop in SSIM), while momentum correction was vital for temporal stability (removing it increased jitter by 39%).

Limitations and Future Work

  • Complex Scenes: While robust, the model sees a modest performance degradation (a 24% drop in CASS) when dealing with complex, multi-object scenes where distracting elements might interfere with the tracking.
  • Segmentation Dependency: The framework relies heavily on the initial quality of the segmentation masks provided by Grounded-SAM2.
  • Future Directions: The authors suggest that exploring more sophisticated noise-space manipulation could further enhance the ability to navigate even more distant semantic shifts.

This summary is based on the paper "MoCA-Video: Motion-Aware Concept Alignment for Consistent Video Editing" under review for ICLR 2026.

Summary updated

Abstract

We present MoCA-Video, a training-free framework for semantic mixing in videos. Operating in the latent space of a frozen video diffusion model, MoCA-Video utilizes class-agnostic segmentation with diagonal denoising scheduler to localize and track the target object across frames. To ensure temporal stability under semantic shifts, we introduce momentum-based correction to approximate novel hybrid distributions beyond trained data distribution, alongside a light gamma residual module that smooths out visual artifacts. We evaluate model's performance using SSIM, LPIPS, and a proposed metric, MoCA-Video, which quantifies semantic alignment between reference and output. Extensive evaluation demonstrates that our model consistently outperforms both training-free and trained baselines, achieving superior semantic mixing and temporal coherence without retraining. Results establish that structured manipulation of diffusion noise trajectories enables controllable and high-quality video editing under semantic shifts.
BibTeX
@misc{zhang2025motion,
  author = {Tong Zhang and Juan C Leon Alcazar and Victor Escorcia and Bernard Ghanem},
  title = {{Motion-Aware Concept Alignment for Consistent Video Editing}},
  year = {2025},
  eprint = {2506.01004},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.01004},
}
Figure