Bilinear Representation for Language-based Image Editing Using Conditional Generative Adversarial Networks

Xiaofeng Mao, Yuefeng Chen, Yuhong Li, Tao Xiong, Yuan He, Hui Xue

In the world of computer vision, Language-Based Image Editing (LBIE) is a bit like digital magic. You show a model a photo of a bird and say, "make the wings blue," and it should happen seamlessly. However, the "magic" behind the scenes—merging the distinct worlds of pixels and natural language—is notoriously difficult.

A team from Alibaba Group recently proposed a more sophisticated way to handle this merger using Bilinear Residual Layers (BRL) within a Conditional Generative Adversarial Network (cGAN) framework. Their paper, published in 2019, demonstrates that by capturing "second-order" interactions between text and images, we can achieve much more realistic and precise edits.

TL;DR

The authors introduce the Bilinear Residual Layer (BRL), a novel conditioning method for GANs that captures complex, second-order correlations between image features and text descriptions. By using a low-rank approximation to keep the model efficient, their approach outperforms standard methods like concatenation and FiLM, producing higher-quality, semantically accurate image edits.

What Problem Does This Paper Address?

The core challenge of LBIE is multimodal fusion. The model must understand which parts of an image correspond to specific words in a description and then modify only those parts while keeping the rest of the image (like the background or the subject's pose) intact.

Existing methods typically use two techniques to combine image and text data:

  1. Concatenation: Simply sticking the text vector onto the image feature vector. This essentially acts as a simple bias and lacks the power to model complex relationships.
  2. Feature-wise Linear Modulation (FiLM): A more advanced method that uses text to scale and shift image features.

The authors argue that these methods are too "linear." They lack the representational power to learn the intricate, non-linear dependencies (second-order correlations) between a specific visual feature and a specific word.

Methods

The researchers propose moving from linear transformations to bilinear transformations.

The Bilinear Residual Layer (BRL)

A bilinear model allows every element of the image feature vector to interact with every element of the text vector. Mathematically, this captures the "second-order" interaction.

However, a full bilinear transformation is computationally "expensive"—it requires a massive number of parameters that grow cubically. To solve this, the authors use two clever tricks:

  • Low-Rank Approximation: They decompose the large weight matrix into smaller, low-rank matrices ($U$ and $V$). This captures the most important interactions without the massive memory overhead.
  • Residual Connections: They add "shortcuts" (similar to ResNet). This allows the model to decide automatically how much of the new "bilinear" information it needs at each layer, making training more stable and allowing the model to preserve original image details when no edit is needed.

The Architecture

The overall system is a cGAN. It consists of:

  • Encoders: Pre-trained models that turn the source image and the text instruction into mathematical vectors.
  • Fusing Module: This is the heart of the paper, containing a stack of $N$ Bilinear Residual Layers.
  • Decoder: Takes the fused representation and generates the final edited image.
  • Discriminator: A "critic" that tries to distinguish between real image-text pairs and the fake ones generated by the model.

Datasets

The model was evaluated on three widely used benchmarks:

  1. Caltech-200 Bird: Images of birds with fine-grained descriptions (e.g., "white with a black superciliary").
  2. Oxford-102 Flower: Images of flowers where color and petal shape are key.
  3. Fashion Synthesis: A dataset for "redressing" people based on descriptions like "the lady wore a white sleeveless dress."

Key Insights and Ideas

  • FiLM is a subset of Bilinear: One of the most interesting theoretical findings in the paper is that FiLM is actually a special, sparse case of a bilinear transformation with a rank of 2 or less. By increasing the rank (e.g., to 64 or 256), the BRL provides a much richer representation.
  • Spatial Awareness: Because the bilinear layer allows for more complex interactions, the model becomes better at "locating" where an edit should happen. For example, in flower editing, the baseline might change the color of the whole image, whereas the BRL model understands that only the petals should change.
  • Preventing Mode Collapse: The authors found that their richer representation helped prevent "mode collapse"—a common GAN failure where the model starts outputting the same blurry, meaningless image regardless of the input.

Key Results

The authors compared their method against a baseline (concatenation) and FiLM using the Inception Score (IS), which measures the quality and diversity of generated images.

Method Caltech Bird (IS) Oxford Flower (IS) Fashion (IS)
Baseline 1.92 5.03 8.65
FiLM 2.59 4.83 8.78
Our (Bil-R256) 2.76 6.26 11.63

Qualitatively, the BRL model (specifically the version with rank 256) produced much sharper images. In the fashion dataset, it successfully changed clothing styles while perfectly preserving the person's identity and pose, whereas previous methods often blurred the person's features.

Limitations and Future Work

While the paper demonstrates a clear leap in quality, there are a few areas for further exploration:

  • Computational Cost: Even with the low-rank trick, bilinear layers are more complex than simple concatenation. The paper notes it takes about a day to converge on a Tesla P100 GPU.
  • Generalization: The authors suggest BRL is a general-purpose layer. Future work could involve testing it in other multimodal tasks like Video Question Answering or Text-to-Image synthesis.
  • Resolution: The experiments were conducted at a $64 \times 64$ resolution. Scaling this to high-definition images would likely require further architectural optimizations.

Summary updated

Abstract

The task of Language-Based Image Editing (LBIE) aims at generating a target image by editing the source image based on the given language description. The main challenge of LBIE is to disentangle the semantics in image and text and then combine them to generate realistic images. Therefore, the editing performance is heavily dependent on the learned representation. In this work, conditional generative adversarial network (cGAN) is utilized for LBIE. We find that existing conditioning methods in cGAN lack of representation power as they cannot learn the second-order correlation between two conditioning vectors. To solve this problem, we propose an improved conditional layer named Bilinear Residual Layer (BRL) to learning more powerful representations for LBIE task. Qualitative and quantitative comparisons demonstrate that our method can generate images with higher quality when compared to previous LBIE techniques.
BibTeX
@inproceedings{mao2019bilinear,
  author = {Xiaofeng Mao and Yuefeng Chen and Yuhong Li and Tao Xiong and Yuan He and Hui Xue},
  title = {{Bilinear Representation for Language-based Image Editing Using Conditional Generative Adversarial Networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=2BkL-YAVdu-},
}

Loading related content...

Figure