Paint by Word

David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, Antonio Torralba

Abstract

We investigate the problem of zero-shot semantic image painting. Instead of painting modifications into an image using only concrete colors or a finite set of semantic concepts, we ask how to create semantic paint based on open full-text descriptions: our goal is to be able to point to a location in a synthesized image and apply an arbitrary new concept such as "rustic" or "opulent" or "happy dog." To do this, our method combines a state-of-the art generative model of realistic images with a state-of-the-art text-image semantic similarity network. We find that, to make large changes, it is important to use non-gradient methods to explore latent space, and it is important to relax the computations of the GAN to target changes to a specific region. We conduct user studies to compare our methods to several baselines.

Introduction

Recent advances in AI have enabled impressive text-to-image generation, where models compose entire scenes from a single prompt. However, semantic painting tools—which allow users to edit specific regions of an image—have typically been limited to fixed vocabularies of predefined classes (e.g., "tree," "sky") or simple color palettes.

"Paint by Word" bridges this gap by introducing zero-shot semantic image painting. This framework allows a user to point to a specific location in an image and apply an open-ended, arbitrary concept using natural language. The goal is to act as a collaborative AI paintbrush, where the user defines the "where" with a mask and the "what" with free-text, while the AI seamlessly blends the new concept into the existing scene.

Methodology

The system achieves localized editing by pairing two separately trained networks: a generative model (StyleGAN2 or BigGAN) to ensure image realism, and the CLIP vision-language model to enforce semantic accuracy.

Overview of the Paint by Word framework showing localized semantic edits for 'Metallic', 'Rustic', 'Purple', and 'Floral' concepts applied to different regions.

Masked Semantic Consistency

The system applies a CLIP-based text-image similarity loss exclusively to the user-painted region (the mask). To prevent the background from changing, an image consistency loss is applied to the area outside the mask.

Spatial Latent Splitting

GANs naturally entangle features; for example, changing a bed's style might inadvertently alter the room's lighting. To solve this, the authors spatially split the GAN's intermediate latent representation. They use one latent vector to freeze the area outside the mask and optimize a separate latent vector for the area inside the mask, forcing a global generator to act locally.

CMA-ES Optimization

Standard gradient descent (Adam) often produces "adversarial examples"—images that score highly in CLIP but appear as unrealistic noise to humans. To maintain visual realism, the authors employ Covariance Matrix Adaptation Evolution Strategy (CMA-ES). This gradient-free sampling method successfully finds latent vectors that match the text while remaining within the GAN's distribution of realistic images.

Key Results and Insights

The method was evaluated across several domains, including CUB Birds, LSUN Bedrooms, and MIT Places.

  • Full Image Generation: In human evaluations, CMA-ES optimization vastly outperformed standard Adam optimization in both accuracy and realism. In the bird domain, it even outperformed DALL-E, with users finding the results more realistic 89% of the time.
  • Localized Editing: A large-scale study of 3,000 edits across 300 bedroom images showed the method is highly effective at altering colors (72.8% accuracy) and textures (46.5% accuracy).
  • Qualitative Success: The model successfully executed complex, abstract edits, such as changing a dog's expression from "sad" to "happy" or turning a standard bed into a "rattan" or "opulent" version.
A 'Rustic' semantic edit applied to the bed, demonstrating the model's ability to alter texture and style while maintaining scene consistency.

Limitations and Future Work

While the method excels at colors and textures, it struggles with fundamental shape alterations (achieving only 31.7% accuracy in user studies). Furthermore, pushing the optimization too hard to match complex prompts can result in visible artifacts or out-of-context styles. Future work involves pairing these techniques with robust GAN inversion to allow for the editing of real-world user photos rather than just GAN-synthesized images.

Summary updated

Abstract

We investigate the problem of zero-shot semantic image painting. Instead of painting modifications into an image using only concrete colors or a finite set of semantic concepts, we ask how to create semantic paint based on open full-text descriptions: our goal is to be able to point to a location in a synthesized image and apply an arbitrary new concept such as "rustic" or "opulent" or "happy dog." To do this, our method combines a state-of-the art generative model of realistic images with a state-of-the-art text-image semantic similarity network. We find that, to make large changes, it is important to use non-gradient methods to explore latent space, and it is important to relax the computations of the GAN to target changes to a specific region. We conduct user studies to compare our methods to several baselines.
BibTeX
@misc{bau2021paint,
  author = {David Bau and Alex Andonian and Audrey Cui and YeonHwan Park and Ali Jahanian and Aude Oliva and Antonio Torralba},
  title = {{Paint by Word}},
  year = {2021},
  eprint = {2103.10951},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.10951},
}
Figure