Motion-Aware Concept Alignment for Consistent Video Editing

Tong Zhang, Juan Carlos León Alcázar, Bernard Ghanem

Introduction

Text-to-video models often struggle with "semantic mixing"—the process of blending features from a reference image into a video object to create entities that do not exist in standard training distributions, such as a "kayak-surfer" or a "cat-astronaut." Existing methods typically fall into two categories: pretrained editing models that maintain stability but often ignore reference images, and training-free blending methods that inject features but suffer from temporal flickering and structural loss. MoCA-Video (Motion-Aware Concept Alignment) addresses these challenges by providing a training-free framework that ensures both semantic alignment and temporal consistency.

A kayak-surfer hybrid generated using MoCA-Video, demonstrating successful semantic mixing of disparate concepts.

Methodology

MoCA-Video operates within the latent space of a frozen video diffusion model, such as VideoCrafter2. The framework is built upon three primary technical pillars:

  1. Latent Space Tracking: The model utilizes Grounded-SAM2 on predicted clean images ($x_0$) at each timestep. An IoU-based maximization algorithm tracks the target object across frames to define a "fusion zone" mask for feature injection.
  2. Momentum-Corrected Denoising: This core innovation treats semantic mixing as a geometric perturbation problem. By calculating the deviation caused by feature injection and applying a momentum term, the model smooths the diffusion trajectory, steering it toward a hybrid distribution that balances the original video structure with new reference features.
  3. Stabilization Techniques: To mitigate visual artifacts like "sparkle," the authors introduce Gamma Residual Stabilization, a light residual noise module that acts as a regularizer. Additionally, a Diagonal Denoising Scheduler allows consecutive frames to share semantic information more effectively by processing them in a queue at varying noise levels.

Experimental Results

The authors evaluated MoCA-Video using the newly created Entity Blending Dataset, which comprises 100 unique entity pairs derived from the CTIB and DAVIS-16 datasets.

A sequence showing a horse-to-unicorn transformation, highlighting the temporal consistency achieved through momentum-corrected denoising.

MoCA-Video demonstrated significant improvements over both training-free and pretrained baselines:

  • Semantic Alignment: The method achieved a Conceptual Alignment Shift Score (CASS) of 4.93, nearly doubling the performance of baselines like TokenFlow-SDEdit (1.98).
  • Temporal Stability: Ablation studies confirmed that momentum correction reduced temporal jitter (measured by LPIPS-T) by 39%.
  • Human Evaluation: In a user study with 20 volunteers, MoCA-Video was rated highest for both blending and overall quality compared to methods like AnimateDiffV2V and RAVE.

Conclusion and Future Work

MoCA-Video provides a robust, training-free solution for consistent video semantic mixing. While the method introduces some computational overhead (approximately 17 seconds per frame on an NVIDIA V100) and faces challenges in highly crowded scenes, it establishes a powerful paradigm for noise-space manipulation. Future work may extend this approach to more complex video-to-video tasks, potentially enabling fully controllable cinematic editing.

Summary updated

Abstract

We present MoCA-Video, a training-free framework for semantic mixing in videos. Operating in the latent space of a frozen video diffusion model, MoCA-Video utilizes class-agnostic segmentation with diagonal denoising scheduler to localize and track the target object across frames. To ensure temporal stability under semantic shifts, we introduce momentum-based correction to approximate novel hybrid distributions beyond trained data distribution, alongside a light gamma residual module that smooths out visual artifacts. We evaluate model's performance using SSIM, LPIPS, and a proposed metric, MoCA-Video, which quantifies semantic alignment between reference and output. Extensive evaluation demonstrates that our model consistently outperforms both training-free and trained baselines, achieving superior semantic mixing and temporal coherence without retraining. Results establish that structured manipulation of diffusion noise trajectories enables controllable and high-quality video editing under semantic shifts.
BibTeX
@misc{zhang2025motion,
  author = {Tong Zhang and Juan Carlos León Alcázar and Bernard Ghanem},
  title = {{Motion-Aware Concept Alignment for Consistent Video Editing}},
  year = {2025},
  eprint = {2506.01004},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.01004},
}
Figure