Introduction
Text-to-video models often struggle with "semantic mixing"—the process of blending features from a reference image into a video object to create entities that do not exist in standard training distributions, such as a "kayak-surfer" or a "cat-astronaut." Existing methods typically fall into two categories: pretrained editing models that maintain stability but often ignore reference images, and training-free blending methods that inject features but suffer from temporal flickering and structural loss. MoCA-Video (Motion-Aware Concept Alignment) addresses these challenges by providing a training-free framework that ensures both semantic alignment and temporal consistency.
Methodology
MoCA-Video operates within the latent space of a frozen video diffusion model, such as VideoCrafter2. The framework is built upon three primary technical pillars:
- Latent Space Tracking: The model utilizes Grounded-SAM2 on predicted clean images ($x_0$) at each timestep. An IoU-based maximization algorithm tracks the target object across frames to define a "fusion zone" mask for feature injection.
- Momentum-Corrected Denoising: This core innovation treats semantic mixing as a geometric perturbation problem. By calculating the deviation caused by feature injection and applying a momentum term, the model smooths the diffusion trajectory, steering it toward a hybrid distribution that balances the original video structure with new reference features.
- Stabilization Techniques: To mitigate visual artifacts like "sparkle," the authors introduce Gamma Residual Stabilization, a light residual noise module that acts as a regularizer. Additionally, a Diagonal Denoising Scheduler allows consecutive frames to share semantic information more effectively by processing them in a queue at varying noise levels.
Experimental Results
The authors evaluated MoCA-Video using the newly created Entity Blending Dataset, which comprises 100 unique entity pairs derived from the CTIB and DAVIS-16 datasets.
MoCA-Video demonstrated significant improvements over both training-free and pretrained baselines:
- Semantic Alignment: The method achieved a Conceptual Alignment Shift Score (CASS) of 4.93, nearly doubling the performance of baselines like TokenFlow-SDEdit (1.98).
- Temporal Stability: Ablation studies confirmed that momentum correction reduced temporal jitter (measured by LPIPS-T) by 39%.
- Human Evaluation: In a user study with 20 volunteers, MoCA-Video was rated highest for both blending and overall quality compared to methods like AnimateDiffV2V and RAVE.
Conclusion and Future Work
MoCA-Video provides a robust, training-free solution for consistent video semantic mixing. While the method introduces some computational overhead (approximately 17 seconds per frame on an NVIDIA V100) and faces challenges in highly crowded scenes, it establishes a powerful paradigm for noise-space manipulation. Future work may extend this approach to more complex video-to-video tasks, potentially enabling fully controllable cinematic editing.
Summary updated