Introduction
Text-driven image-to-image (I2I) translation aims to edit an existing image based on a textual prompt while preserving the original's structural integrity. While large-scale text-to-image (T2I) models like Stable Diffusion have advanced this field, achieving precise control remains challenging. Existing methods often rely on computationally expensive fine-tuning or attention-based mechanisms that struggle to decouple attributes like appearance, layout, and contour.
FBSDiff introduces a training-free, "plug-and-play" approach that operates in the frequency domain. By leveraging the Discrete Cosine Transform (DCT) on internal diffusion features, FBSDiff enables decoupled control over image attributes without requiring model optimization.
Methodology: Frequency Band Substitution
The core of FBSDiff is the Frequency Band Substitution (FBS) layer. Unlike traditional spatial-domain methods, FBSDiff manipulates features in the spectral domain. The process involves three parallel trajectories: Inversion (converting the reference to noise), Reconstruction (extracting clean reference features), and Sampling (generating the target image).
During the initial denoising steps (the calibration phase), the FBS layer intercepts U-Net features and applies a 2D DCT. It then substitutes specific frequency bands from the reference features into the target features:
- Low-frequency bands: Control global appearance, including color and luminance.
- Mid-frequency bands: Isolate layout and structural "bones."
- High-frequency bands: Preserve fine contours and edges.
After substitution, an Inverse DCT (IDCT) returns the features to the spatial domain. This mechanism allows the model to lock in the desired structure early in the generation process, after which the FBS layer is removed to let the model finalize details naturally.
Experimental Evaluation
The authors evaluated FBSDiff using the LAION Aesthetics 6.5+ dataset, creating a benchmark of 1,000 evaluation pairs. The tasks were categorized into "appearance consistency" (preserving style) and "appearance divergence" (changing style while keeping structure).
Quantitatively, FBSDiff demonstrated superior performance in the appearance consistency task, achieving a Structure Similarity score of 0.962 and an LPIPS of 0.241. These metrics indicate that FBSDiff preserves the essence of the reference image more effectively than state-of-the-art competitors like Null-text Inversion.
Key Insights and Discussion
The primary contribution of FBSDiff is the discovery of Spectral Decoupling: semantic attributes in diffusion features are naturally separated by frequency. This allows for explicit control—for instance, preserving only contours while completely changing identity or style.
Furthermore, the method is backbone invariant. As a mathematical wrapper around internal features, it can be integrated into various U-Net-based architectures without modification. It also supports sampling diversity, allowing users to generate multiple variations of an edit by varying the initial seed, a feature often lacking in deterministic inversion-based methods.
Conclusion and Future Work
FBSDiff provides a robust, efficient framework for controllable I2I translation. While highly effective, the method's performance can be sensitive to frequency cut-off hyperparameters, which may require tuning for specific image types. Future research could extend this frequency-domain approach to video editing, leveraging spectral consistency to maintain temporal flow across frames.
Summary updated