Abstract
DeltaEdit is a novel framework that achieves high-quality, text-driven image manipulation without requiring a single word of text during training. By identifying a semantically aligned "CLIP Delta space," the model learns to map visual changes between images to StyleGAN editing directions, allowing it to generalize to unseen text prompts at inference time with remarkable speed and precision.
Introduction
Text-driven image manipulation—changing an image based on a description like "make this person smile" or "turn this into a sunset"—usually suffers from two major bottlenecks: data scarcity and inference inflexibility. Many models require massive datasets of paired images and text descriptions, and current methods often require time-consuming per-prompt optimization. DeltaEdit addresses these issues by identifying a "modality gap" in CLIP: while absolute image and text features are separated, their relative changes (deltas) are remarkably well-aligned.
Methodology: The CLIP Delta Space
The core innovation of DeltaEdit is the CLIP Delta space. The authors discovered that while absolute image and text features are separated by a gap, the difference between two image features ($\Delta i$) is remarkably well-aligned with the difference between two text embeddings ($\Delta t$).
Text-Free Training
The model is trained using only images. It takes two random images, extracts their CLIP visual embeddings, and calculates the delta ($\Delta i$). Simultaneously, it extracts their StyleGAN $S$-space (style space) latent codes. A specialized Delta Mapper neural network then learns to predict the StyleGAN editing direction ($\Delta s$) using the source image's features and the visual delta.
Zero-Shot Inference
At test time, the user provides a source text and a target text. The model calculates the textual delta ($\Delta t$) and plugs it into the mapper in place of $\Delta i$. Because the delta spaces are aligned, the model "knows" how to translate that text change into a StyleGAN manipulation without ever having seen text during training.
Datasets and Evaluation
The authors evaluated DeltaEdit across several popular generative modeling benchmarks to prove its versatility across different domains:
- FFHQ: Human faces (58,000 for training, 12,000 for evaluation).
- LSUN: Categories for Cats, Churches, and Horses.
Key Results
DeltaEdit outperforms existing methods like TediGAN and StyleCLIP across multiple metrics:
- Quality & Consistency: It achieved a superior FID (10.29) and Identity Similarity (0.90), producing realistic images while preserving the original subject's identity.
- Efficiency: Training takes only 2.7 hours, and inference is nearly instantaneous at 73ms per image, compared to the minutes required by optimization-based methods.
- Generalization: The model successfully handles complex, unseen prompts like "bowlcut hairstyle" or "Gothic church" without manual tuning.
Conclusion
By leveraging the alignment of delta spaces in CLIP, DeltaEdit provides a powerful, efficient, and text-free approach to image manipulation. This research suggests that relative changes in embedding spaces are more modality-agnostic than absolute features, opening new doors for cross-modal generative tasks.
Summary updated