Abstract
Reference-based video editing offers a more precise alternative to text-guided methods by allowing users to manually edit a single frame and propagate those changes throughout a sequence. However, existing propagation techniques often suffer from artifacts like color leakage or blurring due to rigid matching strategies. We present AdaptEdit, a framework that utilizes an adaptive correspondence strategy guided by a "proxy-editing" task. By optimizing local attention mechanisms for each token, AdaptEdit achieves significant improvements in fidelity, including a 3.6 dB PSNR gain over state-of-the-art methods on newly introduced benchmarks.
Introduction: The Challenge of Precise Video Editing
As AI-driven video transformation evolves, the limitations of text-guided editing have become apparent. Prompts like "change the car to red" are inherently ambiguous, often leading to inconsistent shades or unintended background alterations. Reference-based editing addresses this by letting users define the exact look on a single "reference" frame using standard tools like Photoshop.
The technical hurdle lies in accurately propagating these edits across the video. Traditional optical flow struggles with non-rigid deformations, while global Cross-Image Attention (CiA) often fails to capture fine geometric details, leading to "color bleeding" or blurry details because it doesn't account for the unique geometric needs of different parts of a frame.
The AdaptEdit Framework
AdaptEdit introduces an adaptive correspondence strategy to refine the diffusion process. The core of the innovation is the Adaptive $k$ Mechanism applied to attention layers.
Adaptive $k$ Mechanism
In standard attention layers, a target pixel matches with $k$ pixels in the reference frame. A fixed $k$ value is problematic: a small $k$ (e.g., $k=1$) preserves sharp edges but can create "black hole" artifacts in uniform areas, while a large $k$ causes blurring and color bleeding. AdaptEdit calculates a unique, optimal $k$ for every individual token (pixel patch), balancing local detail with global consistency.
Proxy-Editing for Optimization
To determine these optimal $k$ values without manual tuning, AdaptEdit employs a proxy-editing task. The system performs a deterministic HSV color shift on the original video. Because this transformation is mathematically ground-truthed, the system can optimize $k$ values to minimize error against this known target. These optimized parameters are then applied to the user's actual, complex edit. To maintain efficiency, this proxy task is executed using only 4 denoising steps, adding minimal computational overhead (roughly 9 seconds per frame).
Experimental Evaluation
The authors evaluated AdaptEdit using three datasets:
- Multi-Reference-Editing (MRE): A standard benchmark for video editing.
- MRE-color: A subset designed for quantitative evaluation using deterministic transforms.
- PVA (Paired Video with Appearance editing): A new dataset rendered in Blender, providing perfect pixel-level ground truth for metrics like PSNR and LPIPS across various motions and environments.
Results and Discussion
AdaptEdit demonstrates superior performance both quantitatively and qualitatively. On the MRE-color dataset, it achieved a PSNR of 26.95, significantly outperforming the CiA baseline (22.18). On the PVA dataset, it showed a 3.6 dB improvement over previous state-of-the-art models.
Qualitatively, the framework handles complex multi-color edits and non-rigid character movements with high temporal consistency. By disentangling the creative edit from the technical propagation, AdaptEdit provides a robust tool for high-fidelity video manipulation.
Conclusion and Future Work
AdaptEdit successfully bridges the gap between user intent and automated video propagation. By leveraging a proxy-editing task to adaptively tune attention, it overcomes the "color leakage" and blurring common in previous methods. While the current proxy task is optimized for appearance, future work may explore more diverse tasks to handle significant structural transformations and further reduce computational overhead.
Summary updated