VALA: Learning Latent Anchors for Training-Free and Temporally Consistent

Zhangkai Wu, Xuhui Fan, Zhongyuan Xie, Kaize Shi, Longbing CaoarXiv 2025 · arXiv · OpenReview

Abstract

Recent advances in training-free video editing have enabled lightweight and precise cross-frame generation by leveraging pre-trained text-to-image diffusion models. However, existing methods often rely on heuristic frame selection to maintain temporal consistency during DDIM inversion, which introduces manual bias and reduces the scalability of end-to-end inference. In this paper, we propose VALA (Variational Alignment for Latent Anchors), a variational alignment module that adaptively selects key frames and compresses their latent features into semantic anchors for consistent video editing. To learn meaningful assignments, VALA proposes a variational framework with a contrastive learning objective. Therefore, it can transform cross-frame latent representations into compressed latent anchors that preserve both content and temporal coherence. Our method can be fully integrated into training-free text-to-image based video editing models. Extensive experiments on real-world video editing benchmarks show that VALA achieves state-of-the-art performance in inversion fidelity, editing quality, and temporal consistency, while offering improved efficiency over prior methods.

Introduction

The advent of powerful text-to-image (T2I) diffusion models has revolutionized visual content creation. Extending these capabilities to video editing, however, remains a significant challenge. The primary hurdle is maintaining temporal consistency—ensuring that objects and backgrounds remain stable across frames without flickering or structural drift. While fine-tuning models on specific videos can achieve this, "training-free" methods are preferred for their efficiency and generalizability.

The Limitations of Heuristic Selection

Most current training-free video editing frameworks, such as TokenFlow or FateZero, rely on manual heuristics to enforce consistency. These methods typically select a fixed set of reference frames (e.g., the first or middle frame) and force all other frames to attend to them during the denoising process.

This approach suffers from two critical drawbacks:

  1. Manual Bias: Fixed reference frames may fail to capture the semantic essence of videos with complex motion or scene changes.
  2. Computational Complexity: Dense cross-frame attention mechanisms often scale quadratically with the number of frames ($O(M^2)$), leading to a "memory explosion" that limits the length and resolution of editable videos.
Conceptual comparison between traditional frame selection and VALA's learned latent anchors.

The VALA Framework

VALA introduces a Variational Alignment module that replaces rigid frame selection with a learned, adaptive system of Latent Anchors. Instead of attending to specific frames, the model learns to compress the entire video's latent features into a compact set of semantic summaries.

Variational Alignment and Contrastive Learning

The core of VALA is a lightweight encoder ($f_\phi$) that maps video tokens to a set of anchors. To ensure these anchors are diverse and representative, the framework employs:

  • Variational Regularization: A KL-divergence loss prevents "mode collapse," where the model might otherwise assign all tokens to a single anchor.
  • Contrastive Objective: This ensures that each anchor remains semantically aligned with the specific visual features it represents.

By performing attention between video frames and these compressed anchors, VALA reduces computational complexity from quadratic to linear ($O(MAC)$), where $A$ is the number of anchors. This allows for faster inference and significantly lower GPU memory usage.

Experimental Evaluation

The researchers evaluated VALA on the DAVIS dataset and a variety of real-world internet videos. The results demonstrate that VALA consistently outperforms state-of-the-art methods in both reconstruction fidelity and editing quality.

Quantitative Performance

VALA achieved a PSNR of 26.18 and an SSIM of 0.875, surpassing competitors like STEM and TokenFlow. More importantly, it significantly improved temporal stability, reducing the Warp Error to 3.8 (compared to 4.9 for TokenFlow and 7.2 for FateZero).

Qualitative Results

Visually, VALA produces edits that are remarkably stable. In a "swan-to-crystal-swan" task, while other methods exhibited flickering or lost the swan's structural integrity, VALA maintained a consistent appearance across the entire sequence.

Qualitative comparison of video editing consistency between VALA and existing methods like FateZero and TokenFlow.

Efficiency Gains

On 640x320 resolution videos, VALA required only 6.2 GB of GPU memory, compared to the 9.4–12 GB required by previous methods. Inference time was also nearly halved, dropping to 0.46 minutes per edit.

Conclusion

VALA represents a significant step forward in training-free video editing. By moving from manual heuristics to a learned variational alignment of latent anchors, it provides a more robust, scalable, and consistent framework for transforming video content. Its ability to integrate seamlessly into existing T2I pipelines makes it a powerful tool for the next generation of AI-driven video editing.

Summary updated

Abstract

Recent advances in training-free video editing have enabled lightweight and precise cross-frame generation by leveraging pre-trained text-to-image diffusion models. However, existing methods often rely on heuristic frame selection to maintain temporal consistency during DDIM inversion, which introduces manual bias and reduces the scalability of end-to-end inference. In this paper, we propose~\textbf{VALA} (\textbf{V}ariational \textbf{A}lignment for \textbf{L}atent \textbf{A}nchors), a variational alignment module that adaptively selects key frames and compresses their latent features into semantic anchors for consistent video editing. To learn meaningful assignments, VALA propose a variational framework with a contrastive learning objective. Therefore, it can transform cross-frame latent representations into compressed latent anchors that preserve both content and temporal coherence. Our method can be fully integrated into training-free text-to-image based video editing models. Extensive experiments on real-world video editing benchmarks show that VALA achieves state-of-the-art performance in inversion fidelity, editing quality, and temporal consistency, while offering improved efficiency over prior methods.
BibTeX
@misc{wu2025vala,
  author = {Zhangkai Wu and Xuhui Fan and Zhongyuan Xie and Kaize Shi and Longbing Cao},
  title = {{VALA: Learning Latent Anchors for Training-Free and Temporally Consistent}},
  year = {2025},
  eprint = {2510.22970},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.22970},
}
Figure