Interactive Image Manipulation with Natural Language Instruction Commands

Seitaro Shinagawa, Koichiro Yoshino, Sakriani Sakti, Yu Suzuki, Satoshi NakamuraarXiv 2018 · arXiv

TL;DR

This paper introduces a framework for interactive image editing that uses natural language instructions to transform a source image into a target image. By modeling the "difference" between images in a shared latent space, the system allows for iterative, controllable modifications like "move the object to the left" or "add glasses to the face."

What Problem Does This Paper Address?

Traditional text-to-image generation (often called "cap2image") typically follows a "one-shot" approach: you provide a full caption, and the model generates an image. However, these models are notoriously difficult to control. If the generated output is almost right but needs a small tweak, the user has to rewrite the entire caption and hope for a better result.

The authors identify a gap in interactivity and fine-grained control. They propose shifting the paradigm from generating an image from a static description to manipulating an existing image using relative instructions. This is not only more intuitive for users but also simplifies the learning task for the model, as instructions describing a change (e.g., "make the hair longer") are much shorter and more focused than full descriptive captions.

Methods

The core of the proposed framework is a neural network that operates in a latent space where images and language are aligned.

  1. Encoder Model: The system uses two encoders. An Image Encoder (ImEnc), based on a DCGAN discriminator architecture, converts the source image into a feature vector ($\phi_{im}$). Simultaneously, an Instruction Encoder (IEnc), using a Long Short-Term Memory (LSTM) network, processes the natural language command into an instruction vector ($\phi_i$).
  2. Latent Transformation: These two vectors are fed into a fully connected (FC) layer. The key hypothesis here is additivity: the model attempts to learn a linear transformation such that {source image vector} + {instruction vector} = {target image vector}.
  3. Decoder Model: The resulting target vector ($\phi_{fc}$) is passed to a Generator (Gen), based on DCGAN, to produce the final manipulated image.
Figure 1: Comparison of natural language conditioned image generation frame- work between cap2image (left) and the proposed framework (right).

To stabilize training, the authors employed feature matching and included auxiliary labels for object class, position, and size, ensuring the generator stayed grounded in the desired attributes.

The network architecture showing how image and instruction vectors are fused to generate a target image

Datasets

The authors evaluated their framework on two distinct datasets:

  • Artificial MNIST: A synthetic dataset created by placing MNIST digits on a larger canvas. It contains 3.69 million triplets (source image, instruction, target image) focusing on operations like "move," "expand," "compress," and "remove" across different positions and sizes.
  • Avatar Dataset: A more "natural" task using 8,000 image pairs from AvatarMaker.com. Instructions were collected via crowdsourcing to describe attribute changes, such as "make his chin and jaw fuller" or "put a beard."

Key Insights and Ideas

  • Instruction vs. Caption: Instructions are semantically denser regarding change. Because they only describe the delta between two states, the language encoder can focus on verbs and directional cues rather than exhaustive descriptions.
  • Semantic Vector Space: Visualization of the instruction vectors revealed that the model successfully learned conceptual relationships. For instance, "expand" and "compress" were learned as inverse operations in the latent space.
  • Emergent Generalization: In the avatar experiments, the model was able to generate a "woman with a beard" when instructed, even though such an image did not exist in the training set. This suggests the model decoupled the concept of "beard" from the "male" attribute.

Key Results

  • MNIST Performance: The model achieved high structural similarity (SSIM) scores, successfully translating digits according to directional commands.
  • Subjective Quality: In human evaluations of the avatar task, 45.3% of the generated images were rated as "very similar" to the gold-standard target images.
  • Interpolation: By varying the coefficient of the instruction vector, the authors could smoothly transition an image (e.g., gradually lengthening hair), demonstrating that the latent space is well-structured for continuous manipulation.

Limitations and Future Work

Despite the successes, the authors noted several hurdles:

  • Granularity Issues: While large changes (like adding glasses) worked well, subtle changes to facial features (mouth, nose, ears) were difficult for the model to capture accurately.
  • Word Dominance: In some cases, a specific noun (like "glasses") would overpower the verb (like "remove"), causing the model to add glasses even when told to take them away.
  • Dataset Constraints: The model struggled with "unseen" logic, such as moving an object off the canvas, likely because the training data lacked examples of objects disappearing via movement.

Future work aims to scale this framework to more complex, real-world datasets and a wider variety of image manipulations.

Summary updated

Abstract

We propose an interactive image-manipulation system with natural language instruction, which can generate a target image from a source image and an instruction that describes the difference between the source and the target image. The system makes it possible to modify a generated image interactively and make natural language conditioned image generation more controllable. We construct a neural network that handles image vectors in latent space to transform the source vector to the target vector by using the vector of instruction. The experimental results indicate that the proposed framework successfully generates the target image by using a source image and an instruction on manipulation in our dataset.
BibTeX
@misc{shinagawa2018interactive,
  author = {Seitaro Shinagawa and Koichiro Yoshino and Sakriani Sakti and Yu Suzuki and Satoshi Nakamura},
  title = {{Interactive Image Manipulation with Natural Language Instruction Commands}},
  year = {2018},
  eprint = {1802.08645},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1802.08645},
}
Figure