AdaptEdit: An Adaptive Correspondence Guidance Framework for Reference-Based Video Editing

Tongtong Su, Chengyu Wang, Bingyan Liu, Jun Huang, Dongming Lu

Abstract

Reference-based video editing offers a more precise alternative to text-guided methods by allowing users to manually edit a single frame and propagate those changes throughout a sequence. However, existing propagation techniques often suffer from artifacts like color leakage or blurring due to rigid matching strategies. We present AdaptEdit, a framework that utilizes an adaptive correspondence strategy guided by a "proxy-editing" task. By optimizing local attention mechanisms for each token, AdaptEdit achieves significant improvements in fidelity, including a 3.6 dB PSNR gain over state-of-the-art methods on newly introduced benchmarks.

Introduction: The Challenge of Precise Video Editing

As AI-driven video transformation evolves, the limitations of text-guided editing have become apparent. Prompts like "change the car to red" are inherently ambiguous, often leading to inconsistent shades or unintended background alterations. Reference-based editing addresses this by letting users define the exact look on a single "reference" frame using standard tools like Photoshop.

The technical hurdle lies in accurately propagating these edits across the video. Traditional optical flow struggles with non-rigid deformations, while global Cross-Image Attention (CiA) often fails to capture fine geometric details, leading to "color bleeding" or blurry details because it doesn't account for the unique geometric needs of different parts of a frame.

Comparison between text-guided video editing and AdaptEdit's reference-based approach, highlighting the reduction in ambiguity and artifacts.

The AdaptEdit Framework

AdaptEdit introduces an adaptive correspondence strategy to refine the diffusion process. The core of the innovation is the Adaptive $k$ Mechanism applied to attention layers.

Adaptive $k$ Mechanism

In standard attention layers, a target pixel matches with $k$ pixels in the reference frame. A fixed $k$ value is problematic: a small $k$ (e.g., $k=1$) preserves sharp edges but can create "black hole" artifacts in uniform areas, while a large $k$ causes blurring and color bleeding. AdaptEdit calculates a unique, optimal $k$ for every individual token (pixel patch), balancing local detail with global consistency.

Proxy-Editing for Optimization

To determine these optimal $k$ values without manual tuning, AdaptEdit employs a proxy-editing task. The system performs a deterministic HSV color shift on the original video. Because this transformation is mathematically ground-truthed, the system can optimize $k$ values to minimize error against this known target. These optimized parameters are then applied to the user's actual, complex edit. To maintain efficiency, this proxy task is executed using only 4 denoising steps, adding minimal computational overhead (roughly 9 seconds per frame).

Experimental Evaluation

The authors evaluated AdaptEdit using three datasets:

  1. Multi-Reference-Editing (MRE): A standard benchmark for video editing.
  2. MRE-color: A subset designed for quantitative evaluation using deterministic transforms.
  3. PVA (Paired Video with Appearance editing): A new dataset rendered in Blender, providing perfect pixel-level ground truth for metrics like PSNR and LPIPS across various motions and environments.

Results and Discussion

AdaptEdit demonstrates superior performance both quantitatively and qualitatively. On the MRE-color dataset, it achieved a PSNR of 26.95, significantly outperforming the CiA baseline (22.18). On the PVA dataset, it showed a 3.6 dB improvement over previous state-of-the-art models.

Qualitative results across various domains including character animation and object appearance editing, demonstrating temporal consistency and detail preservation.

Qualitatively, the framework handles complex multi-color edits and non-rigid character movements with high temporal consistency. By disentangling the creative edit from the technical propagation, AdaptEdit provides a robust tool for high-fidelity video manipulation.

Conclusion and Future Work

AdaptEdit successfully bridges the gap between user intent and automated video propagation. By leveraging a proxy-editing task to adaptively tune attention, it overcomes the "color leakage" and blurring common in previous methods. While the current proxy task is optimized for appearance, future work may explore more diverse tasks to handle significant structural transformations and further reduce computational overhead.

Summary updated

Abstract

Video editing is a pivotal process for customizing video content according to user needs. However, existing text-guided methods often lead to ambiguities regarding user intentions and restrict fine-grained control for editing specific aspects in videos. To overcome these limitations, this paper introduces a novel approach named \emph{AdaptEdit}, which focuses on reference-based video editing that disentangles the editing process. It achieves this by first editing a reference image and then adaptively propagating its appearance across other frames to complete the video editing. While previous propagation methods, such as optical flow and the temporal modules of recent video generative models, struggle with object deformations and large motions, we propose an adaptive correspondence strategy that accurately transfers the appearance from the reference frame to the target frames by leveraging inter-frame semantic correspondences in the original video. By implementing a proxy-editing task to optimize hyperparameters for image token-level correspondence, our method effectively balances the need to maintain the target frame's structure while preventing leakage of irrelevant appearance. To more accurately evaluate editing beyond the semantic-level consistency provided by CLIP-style models, we introduce a new dataset, PVA, which supports pixel-level evaluation. Our method outperforms the best-performing baseline with a clear PSNR improvement of 3.6 dB.
BibTeX
@inproceedings{su2025adaptedit,
  author = {Tongtong Su and Chengyu Wang and Bingyan Liu and Jun Huang and Dongming Lu},
  title = {{AdaptEdit: An Adaptive Correspondence Guidance Framework for Reference-Based Video Editing}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/1131},
}
Figure