Language-Guided Global Image Editing via Cross-Modal Cyclic Mechanism

Wentao Jiang, Ning Xu, Jiayun Wang, Chen Gao, Jing Shi, Zhe Lin, Si Liu

Introduction

Language-guided image editing aims to democratize professional-grade photo manipulation by allowing users to describe desired changes in natural language. While tools like Photoshop require technical expertise, a language-based interface allows for intuitive requests such as "make the sunset more vibrant" or "reduce the glare." However, developing robust models for this task is challenging due to significant data imbalances in existing datasets and the need for spatial-adaptive adjustments rather than uniform global filters.

Overview of the language-guided global image editing task, illustrating the transformation from an input image to a target based on a text request.

Challenges in Global Image Editing

Existing research in this field faces two primary hurdles:

  1. Data Scarcity and Imbalance: Real-world datasets are often small and heavily biased. For instance, a large majority of user requests involve increasing brightness, while more complex edits like hue or saturation adjustments are rare. This leads to models that default to brightening images regardless of the specific prompt.
  2. Uniformity vs. Adaptivity: Standard global editing tools often apply a single transformation across the entire image. However, a request to "brighten the photo" typically implies a need to lift shadows in dark regions without overexposing already bright areas.

Methodology: The CAGAN Framework

To address these issues, the authors propose CAGAN (Cycle Augmentation GAN), which incorporates a cross-modal cyclic mechanism and spatial-adaptive attention.

Cross-Modal Cyclic Mechanism & EDNet

CAGAN introduces the Editing Description Network (EDNet). While the primary Generator (G) produces an edited image from an input and a text request, EDNet performs the inverse: it analyzes a pair of images (before and after) to predict the "editing embedding" that describes the transformation. This creates a self-supervised cycle, allowing the model to learn from "free" data and become more robust to the noisy text labels found in human-annotated datasets.

Image-Request Attention (IRA)

To achieve spatial-adaptive editing, CAGAN utilizes the Image-Request Attention (IRA) module. This module calculates the correlation between the text request and specific image patches, generating a spatial attention map. This map guides the generator to apply different degrees of editing to different regions, ensuring that adjustments are contextually appropriate.

Data Augmentation

The framework employs "swapping augmentation"—treating the target image as the input to learn inverse operations—and synthetic augmentations to populate rare editing categories like hue and saturation, mitigating the effects of dataset bias.

Evaluation Metrics

The authors argue that standard pixel-level metrics like L1 loss are insufficient for evaluating creative edits, as multiple "correct" versions of a warmer or moodier image can exist. Instead, they propose the Redescription Similarity Score (RSS). This metric uses a pre-trained "speaker" model to describe the edited image; if the speaker's description matches the original user request, the edit is deemed successful.

Results and Discussion

CAGAN was evaluated on the GIER and MA5k-Req benchmarks, significantly outperforming baselines such as TAGAN and ManiGAN. On the GIER dataset, CAGAN achieved an RSS (CIDEr) score of 37.03, a substantial improvement over the next best baseline (30.16).

Qualitative comparison showing CAGAN's ability to perform complex color and tone adjustments on both synthetic and real-world images.

Qualitative user studies further confirmed that humans prefer CAGAN's results, particularly for complex requests like "remove the purple tone" or "fix overexposure," where other models often fail or introduce artifacts.

Conclusion

CAGAN represents a significant step forward in language-guided global image editing by leveraging cyclic self-supervision and spatial attention. While the current framework focuses on global adjustments, future work may extend these mechanisms to local object manipulation and multi-step editing tasks.

Summary updated

Abstract

Editing an image automatically via a linguistic request can significantly save laborious manual work and is friendly to photography novice. In this paper, we focus on the task of language-guided global image editing. Existing works suffer from imbalanced and insufficient data distribution of real-world datasets and thus fail to understand language requests well. To handle this issue, we propose to create a cycle with our image generator by creating a novel model called Editing Description Network (EDNet) which predicts an editing embedding given a pair of images. Given the cycle, we propose several free augmentation strategies to help our model understand various editing requests given the imbalanced dataset. In addition, two other novel ideas are proposed: an Image-Request Attention (IRA) module which allows our method to edit an image spatial-adaptively when the image requires different editing degree at different regions, as well as a new evaluation metric for this task which is more semantic and reasonable than conventional pixel losses (e.g. L1). Extensive experiments on two benchmark datasets demonstrate the effectiveness of our method over existing approaches.
BibTeX
@inproceedings{jiang2021language,
  author = {Wentao Jiang and Ning Xu and Jiayun Wang and Chen Gao and Jing Shi and Zhe Lin and Si Liu},
  title = {{Language-Guided Global Image Editing via Cross-Modal Cyclic Mechanism}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00212},
}
Figure