SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models

Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, Ying Shan

Introduction

Instruction-based image editing has advanced significantly with models like InstructPix2Pix. However, these models often hit a "reasoning ceiling." While they excel at simple requests like "make the sky blue," they struggle with complex instructions requiring spatial reasoning (e.g., "change the middle apple to an orange") or world knowledge (e.g., "remove the object that tells the time").

SmartEdit handles a variety of complex editing tasks involving spatial reasoning, mirror logic, and world knowledge.

The authors identify two primary limitations in existing frameworks that cause this failure:

  1. The Encoder Gap: Reliance on CLIP text encoders, which are excellent at capturing global styles but lack fine-grained spatial and semantic reasoning capabilities.
  2. One-Way Interaction: A lack of bidirectional communication between textual instructions and visual features during the diffusion process, preventing the model from grounding instructions in the actual pixel layout.

Methodology: SmartEdit Architecture

SmartEdit addresses these gaps by integrating a Multimodal Large Language Model (MLLM) into the diffusion pipeline, replacing the standard text encoder with a reasoning engine.

MLLM as the Reasoning Brain

Instead of a standard CLIP encoder, SmartEdit utilizes LLaVA. By appending special [IMG] tokens to the text instructions, the model extracts hidden states that encapsulate both linguistic context and visual-reasoning data. This allows the model to "understand" the scene before attempting to edit it.

Bidirectional Interaction Module (BIM)

The core innovation is the BIM, which replaces simple one-way cross-attention. It facilitates a "dialogue" between image features from the encoder and text features from the MLLM. This ensures that the model's understanding of the instruction is grounded in the specific spatial layout of the input image, allowing for precise object targeting.

Training Strategy

The system is trained in two stages:

  • Alignment: Using a QFormer to align MLLM outputs with the diffusion model's latent space.
  • Fine-tuning: Applying LoRA to the MLLM while training on a mix of standard editing data, perception data (segmentation), and a small set of complex synthetic samples.

Datasets and the Reason-Edit Benchmark

To "activate" the MLLM's reasoning for editing, the authors generated 476 high-quality synthetic samples covering spatial relations and mirror reflections. They also introduced Reason-Edit, a new benchmark of 219 image-text pairs designed to evaluate complex understanding and reasoning in image manipulation.

Key Results

SmartEdit significantly outperforms state-of-the-art models like InstructDiffusion and MagicBrush on complex tasks.

SmartEdit demonstrates precise spatial reasoning, accurately identifying and modifying the left, middle, or right object as specified.
  • Quantitative Performance: On the Reason-Edit benchmark, SmartEdit-13B achieved an Instruction Alignment (Ins-align) score of 0.817, nearly doubling the performance of InstructDiffusion (0.483).
  • Qualitative Success: The model successfully handles "mirror" logic—changing a reflection without altering the real object—and tasks requiring world knowledge, such as identifying a "tool used to cut cake" and removing it.
  • Human Evaluation: In user studies, over 67% of participants preferred SmartEdit’s results for their superior alignment with complex instructions compared to existing methods.

Conclusion

SmartEdit demonstrates that the bottleneck in complex image editing is often the text encoder rather than the diffusion process itself. By leveraging the reasoning power of MLLMs and a bidirectional interaction module, SmartEdit sets a new standard for instruction-based image manipulation. While the model is computationally heavier than CLIP-based editors, it provides the necessary reasoning capabilities for the next generation of creative tools.

Summary updated

Abstract

Current instruction-based editing methods, such as InstructPix2Pix, often fail to produce satisfactory results in complex scenarios due to their dependence on the simple CLIP text encoder in diffusion models. To rectify this, this paper introduces SmartEdit, a novel approach to instruction-based image editing that leverages Multimodal Large Language Models (MLLMs) to enhance their understanding and reasoning capabilities. However, direct integration of these elements still faces challenges in situations requiring complex reasoning. To mitigate this, we propose a Bidirectional Interaction Module that enables comprehensive bidirectional information interactions between the input image and the MLLM output. During training, we initially incorporate perception data to boost the perception and understanding capabilities of diffusion models. Subsequently, we demonstrate that a small amount of complex instruction editing data can effectively stimulate SmartEdit's editing capabilities for more complex instructions. We further construct a new evaluation dataset, Reason-Edit, specifically tailored for complex instruction-based image editing. Both quantitative and qualitative results on this evaluation dataset indicate that our SmartEdit surpasses previous methods, paving the way for the practical application of complex instruction-based image editing.
BibTeX
@inproceedings{huang2023smartedit,
  author = {Yuzhou Huang and Liangbin Xie and Xintao Wang and Ziyang Yuan and Xiaodong Cun and Yixiao Ge and Jiantao Zhou and Chao Dong and Rui Huang and Ruimao Zhang and Ying Shan},
  title = {{SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52733.2024.00799},
}
Figure