Bilinear Representation for Language-based Image Editing Using Conditional Generative Adversarial Networks

Xiaofeng Mao, Yuefeng Chen, Yuhong Li, Tao Xiong, Yuan He, Hui Xue

Abstract

The task of Language-Based Image Editing (LBIE) aims to generate a target image by modifying a source image according to a natural language description. A primary challenge in LBIE is effectively disentangling and then fusing the semantics of images and text to generate realistic results. In this work, we utilize a conditional generative adversarial network (cGAN) and identify that existing conditioning methods lack sufficient representation power as they fail to capture second-order correlations between conditioning vectors. To address this, we propose the Bilinear Residual Layer (BRL). Qualitative and quantitative evaluations demonstrate that our method produces higher-quality images compared to previous LBIE techniques.

The Challenge of Multimodal Fusion

LBIE requires a model to understand which visual features correspond to specific words and then modify only those features while preserving the rest of the image context, such as background or pose. Traditional methods for combining these modalities often rely on simple concatenation or Feature-wise Linear Modulation (FiLM).

However, these approaches are fundamentally linear. Concatenation acts as a simple bias, while FiLM uses text to scale and shift image features. The authors argue that these methods lack the representational power to learn intricate, non-linear dependencies—specifically second-order correlations—between visual features and linguistic descriptors.

Methodology: Bilinear Residual Layers

To overcome the limitations of linear fusion, this research introduces the Bilinear Residual Layer (BRL). A bilinear model allows every element of the image feature vector to interact with every element of the text vector, capturing complex second-order interactions.

The proposed cGAN architecture featuring the Fusing Module with Bilinear Residual Layers (BRL) and the internal structure of a BRL unit.

To manage the computational complexity and high parameter count of full bilinear transformations, the authors implemented two key innovations:

  1. Low-Rank Approximation: The large weight matrix is decomposed into smaller, low-rank matrices ($U$ and $V$), capturing essential interactions without massive memory overhead.
  2. Residual Connections: By adding "shortcuts," the model can automatically determine the necessary amount of bilinear information at each layer. This stabilizes training and helps preserve original image details when no edit is required.

An important theoretical finding of this work is that FiLM is actually a sparse, low-rank (rank $\le 2$) case of a bilinear transformation. By increasing the rank (e.g., to 256), the BRL provides a significantly richer representation.

Experimental Results

The model was evaluated on three benchmarks: Caltech-200 Bird, Oxford-102 Flower, and Fashion Synthesis. Quantitative results using the Inception Score (IS) show that the Bilinear approach (Bil-R256) consistently outperforms both the baseline and FiLM.

Method Caltech Bird (IS) Oxford Flower (IS) Fashion (IS)
Baseline 1.92 5.03 8.65
FiLM 2.59 4.83 8.78
Our (Bil-R256) 2.76 6.26 11.63

Qualitatively, the BRL model produces sharper images with better semantic alignment. For instance, in flower editing, the model correctly identifies that only the petals should change color, whereas baseline models often shift the color of the entire image.

Qualitative comparison across three datasets showing that the Bilinear method (Bil-R256) produces more accurate and sharper edits than Baseline or FiLM methods.

Conclusion

By introducing Bilinear Residual Layers into the cGAN framework, this work demonstrates a significant improvement in the quality and precision of language-based image editing. The use of low-rank approximations makes these powerful second-order interactions computationally feasible, paving the way for more sophisticated multimodal generative models.

Summary updated

Abstract

The task of Language-Based Image Editing (LBIE) aims at generating a target image by editing the source image based on the given language description. The main challenge of LBIE is to disentangle the semantics in image and text and then combine them to generate realistic images. Therefore, the editing performance is heavily dependent on the learned representation. In this work, conditional generative adversarial network (cGAN) is utilized for LBIE. We find that existing conditioning methods in cGAN lack of representation power as they cannot learn the second-order correlation between two conditioning vectors. To solve this problem, we propose an improved conditional layer named Bilinear Residual Layer (BRL) to learning more powerful representations for LBIE task. Qualitative and quantitative comparisons demonstrate that our method can generate images with higher quality when compared to previous LBIE techniques.
BibTeX
@inproceedings{mao2019bilinear,
  author = {Xiaofeng Mao and Yuefeng Chen and Yuhong Li and Tao Xiong and Yuan He and Hui Xue},
  title = {{Bilinear Representation for Language-based Image Editing Using Conditional Generative Adversarial Networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=2BkL-YAVdu-},
}
Figure