Describe What to Change: A Text-guided Unsupervised Image-to-Image Translation Approach

Yahui Liu, Marco De Nadai, Deng Cai, Huayang Li, Xavier Alameda-Pineda, Nicu Sebe, Bruno Lepri

Abstract

Image manipulation via natural language often requires exhaustive descriptions of the entire scene, which is counterintuitive for simple editing tasks. We present DWC-GAN, a novel unsupervised framework that allows users to modify images using concise, delta-based text commands (e.g., "make him smile"). By disentangling image content from attributes and modeling the attribute space using a Gaussian Mixture Model (GMM), DWC-GAN addresses linguistic ambiguity and produces diverse, high-quality results. Our approach eliminates the need for paired data or human-annotated captions through an automatic text protocol. Experimental results on the CelebA and CUB Birds datasets show that DWC-GAN significantly outperforms state-of-the-art baselines in both Fréchet Inception Distance (FID) and human-perceived realism.

Introduction

The goal of text-guided image manipulation is to provide a seamless interface for visual editing. However, current models face significant challenges: they often require "description overload" (detailing every aspect of the image), suffer from data scarcity (requiring paired images or rich captions), and struggle with the inherent ambiguity of human language.

To address these issues, we introduce the Describe What to Change (DWC) paradigm. Instead of describing the desired final state, users provide commands that specify only the intended modifications. This approach is implemented via DWC-GAN, an unsupervised, multi-modal translation model.

Methodology

DWC-GAN treats image manipulation as a translation between visual domains. The architecture relies on several key components:

  1. Disentangled Representation: Dual encoders separate the input image into a fixed content code (preserving identity/structure) and a latent attribute code (representing features to be modified).
  2. GMM Attribute Space: Attributes are modeled using a Gaussian Mixture Model (GMM). Each component represents a specific domain, allowing the model to sample diverse points and generate multi-modal outputs for a single command.
  3. Text-to-Attribute Mapping: An RNN processes the text command and the current attribute state to navigate the GMM toward a target attribute state.
  4. AdaIN Injection and Unsupervised Attention: Target attributes are injected into the generator via Adaptive Instance Normalization (AdaIN). An automatically learned attention mask ensures that only relevant pixels are modified, preserving the background and unrelated features.

Experimental Setup

The model was evaluated on two benchmarks:

  • CelebA: 200,000+ face images with 8 target attributes.
  • CUB Birds: ~12,000 bird images focusing on fine-grained color changes.

To enable unsupervised training, we utilized an Automatic Text Protocol that generates training commands from existing attribute labels, bypassing the need for manual captioning.

Results and Discussion

DWC-GAN demonstrates superior performance across multiple metrics. On the CelebA dataset, it achieved an FID of 32.14, significantly lower than StarGAN* (50.66) and TAGAN (421.84), indicating higher realism.

Facial attribute manipulation results on the CelebA dataset.

In human evaluations, 82% of DWC-GAN's outputs were rated as semantically correct, compared to 25.6% for StarGAN*. The model also excels in generating diverse variations for ambiguous commands and supports zero-shot composition of attributes.

Fine-grained color manipulation results on the CUB Birds dataset.

Conclusion

DWC-GAN shifts the image editing paradigm from full-scene description to targeted commands. By embracing linguistic uncertainty through a GMM-based attribute space, the model provides a more flexible and intuitive tool for image translation. Future work will focus on expanding the attribute scope and handling more complex, naturalistic language inputs.

Summary updated

Abstract

Manipulating visual attributes of images through human-written text is a very challenging task. On the one hand, models have to learn the manipulation without the ground truth of the desired output. On the other hand, models have to deal with the inherent ambiguity of natural language. Previous research usually requires either the user to describe all the characteristics of the desired image or to use richly-annotated image captioning datasets. In this work, we propose a novel unsupervised approach, based on image-to-image translation, that alters the attributes of a given image through a command-like sentence such as "change the hair color to black". Contrarily to state-of-the-art approaches, our model does not require a human-annotated dataset nor a textual description of all the attributes of the desired image, but only those that have to be modified. Our proposed model disentangles the image content from the visual attributes, and it learns to modify the latter using the textual description, before generating a new image from the content and the modified attribute representation. Because text might be inherently ambiguous (blond hair may refer to different shadows of blond, e.g. golden, icy, sandy), our method generates multiple stochastic versions of the same translation. Experiments show that the proposed model achieves promising performances on two large-scale public datasets: CelebA and CUB. We believe our approach will pave the way to new avenues of research combining textual and speech commands with visual attributes.
BibTeX
@inproceedings{liu2020describe,
  author = {Yahui Liu and Marco De Nadai and Deng Cai and Huayang Li and Xavier Alameda-Pineda and Nicu Sebe and Bruno Lepri},
  title = {{Describe What to Change: A Text-guided Unsupervised Image-to-Image Translation Approach}},
  booktitle = {ACM Multimedia},
  year = {2020},
  doi = {10.1145/3394171.3413505},
}
Figure