Doubly Abductive Counterfactual Inference for Text-based Image Editing

Xue Song, Jiequan Cui, Hanwang Zhang, Jingjing Chen, Richang Hong, Yu-Gang Jiang

Introduction

Text-based image editing (TBIE) aims to modify a source image according to a natural language prompt while preserving the original content's structural and stylistic integrity. This task is inherently challenging because it is "zero-shot"—there is no paired training data of "before" and "after" images. Current state-of-the-art methods often rely on fine-tuning large-scale generative models like Stable Diffusion on a single image. However, these methods frequently struggle with a fundamental trade-off: insufficient fine-tuning leads to low fidelity (loss of original details), while excessive fine-tuning results in overfitting, where the model ignores the editing prompt entirely.

The DAC Framework: A Causal Approach

To address this, researchers have introduced the Doubly Abductive Counterfactual inference (DAC) framework. DAC reinterprets TBIE as a causal reasoning problem. In this view, the source image $I$ is seen as being generated by a prompt $P$ and some latent "exogenous" variables that capture the image's unique details. The goal is to determine what the image $I'$ would look like if the prompt were changed to $P'$.

The framework employs a two-stage "Abduction" process to capture both the image's identity and the desired semantic change:

  1. Abduction-1 (Fidelity): A UNet LoRA (Low-Rank Adaptation) is used to encode the fine-grained details of the source image, ensuring the model maintains high fidelity to the original structure.
  2. Abduction-2 (Editability): To prevent the overfitting common in single-image fine-tuning, a second exogenous variable—a Text Encoder LoRA—is introduced. This step infers the semantic transition from the edited state back to the original state.
Causal diagrams illustrating the Abduction and Action & Prediction phases of the DAC framework.

By reversing the second LoRA (the "Action" phase), the model can steer the generation toward the counterfactual state $I'$ (the "Prediction" phase) without sacrificing the details captured in the first step.

Experimental Evaluation

The DAC framework was evaluated using diverse datasets, including high-resolution Unsplash images and the InstructPix2Pix benchmark. The researchers also developed a custom benchmark covering six editing categories: addition, removal, manipulation, replacement, style transfer, and face manipulation.

Example of face manipulation using DAC: transitioning from "A woman" to "A woman in a big smile."

In a user study involving 110 participants, DAC was preferred 75.3% of the time over competing methods like Imagic and SINE. Quantitatively, DAC achieved a superior balance between LPIPS (fidelity) and CLIP-score (editability) on the InstructPix2Pix benchmark.

Limitations and Future Work

Despite its performance, DAC has limitations. The dual optimization phases make it computationally intensive, requiring approximately 15 minutes per edit on an A100 GPU. Additionally, while it supports sequential edits, image quality tends to degrade over multiple turns. Future research will focus on integrating faster models, such as Latent Consistency Models, to reduce inference time and improving the model's "common sense" reasoning by leveraging more robust base models.

Summary updated

Abstract

We study text-based image editing (TBIE) of a single image by counterfactual inference because it is an elegant formulation to precisely address the requirement: the edited image should retain the fidelity of the original one. Through the lens of the formulation, we find that the crux of TBIE is that existing techniques hardly achieve a good trade-off between editability and fidelity, mainly due to the overfitting of the single-image fine-tuning. To this end, we propose a Doubly Abductive Counterfactual inference framework (DAC). We first parameterize an exogenous variable as a UNet LoRA, whose abduction can encode all the image details. Second, we abduct another exogenous variable parameterized by a text encoder LoRA, which recovers the lost editability caused by the overfitted first abduction. Thanks to the second abduction, which exclusively encodes the visual transition from post-edit to pre-edit, its inversion -- subtracting the LoRA -- effectively reverts pre-edit back to post-edit, thereby accomplishing the edit. Through extensive experiments, our DAC achieves a good trade-off between editability and fidelity. Thus, we can support a wide spectrum of user editing intents, including addition, removal, manipulation, replacement, style transfer, and facial change, which are extensively validated in both qualitative and quantitative evaluations. Codes are in https://github.com/xuesong39/DAC.
BibTeX
@inproceedings{song2024doubly,
  author = {Xue Song and Jiequan Cui and Hanwang Zhang and Jingjing Chen and Richang Hong and Yu-Gang Jiang},
  title = {{Doubly Abductive Counterfactual Inference for Text-based Image Editing}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure