Variational Hetero-Encoder Randomized Generative Adversarial Networks for Joint Image-Text Modeling

Hao Zhang, Bo Chen, Long Tian, Zhengjue Wang, Mingyuan Zhou

Abstract

For bidirectional joint image-text modeling, we develop variational hetero-encoder (VHE) randomized generative adversarial network (GAN), a versatile deep generative model that integrates a probabilistic text decoder, probabilistic image encoder, and GAN into a coherent end-to-end multi-modality learning framework. VHE randomized GAN (VHE-GAN) encodes an image to decode its associated text, and feeds the variational posterior as the source of randomness into the GAN image generator. We plug three off-the-shelf modules, including a deep topic model, a ladder-structured image encoder, and StackGAN++, into VHE-GAN, which already achieves competitive performance. This further motivates the development of VHE-raster-scan-GAN that generates photo-realistic images in not only a multi-scale low-to-high-resolution manner, but also a hierarchical-semantic coarse-to-fine fashion. By capturing and relating hierarchical semantic and visual concepts with end-to-end training, VHE-raster-scan-GAN achieves state-of-the-art performance in a wide variety of image-text multi-modality learning and generation tasks.

Introduction

In the realm of multi-modal AI, most deep learning models are strictly unidirectional—they either generate images from text (like standard text-to-image GANs) or text from images (like image captioning models). While some bidirectional probabilistic graphical models exist, they often rely on restrictive parametric assumptions that limit their ability to generate high-quality, photo-realistic images.

Furthermore, existing text-to-image GANs often separate text feature extraction from image generation, or they rely on sequence models (like RNNs) that struggle to capture macro-level global context. As a result, these models frequently "forget" or misrepresent fine-grained semantic details provided in the text prompt. The VHE-GAN framework addresses these challenges by building a unified, end-to-end model that can bidirectionally translate between images and text while faithfully preserving complex semantic details.

Methodology

The proposed VHE-GAN framework merges Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) across modalities.

Variational Hetero-Encoder (VHE)

Instead of a standard VAE that encodes and decodes the same modality, the VHE encodes an image to decode its associated text. The variational posterior from the VHE is fed into the GAN's image generator as the source of randomness, replacing the standard Gaussian noise. This forces the VHE and GAN to share a meaningful latent space, which helps the GAN resist mode collapse and the VHE learn a richer representation.

Overview of the hierarchical text decoder (a) and the baseline VHE-StackGAN++ architecture (b).

VHE-raster-scan-GAN

To fully exploit the hierarchical nature of deep topic models—which naturally organize text into a multi-layer hierarchy of topics from broad concepts to specific details—the authors designed the "raster-scan" GAN. This generator creates images not just by scaling up resolution, but in a hierarchical-semantic coarse-to-fine fashion. Top layers generate general shapes and backgrounds based on broad text topics, while bottom layers refine specific textures and colors based on fine-grained text topics.

The VHE-raster-scan-GAN architecture (c) illustrating the alignment of hierarchical semantic topics with multi-scale visual generation.

Key Insights

  • Topic Models for Vision: Deep topic models (like the Poisson gamma belief network) are often superior to sequence models for capturing macro-level visual concepts. They handle diverse text formats, including tags and long documents, more robustly than RNNs.
  • Semantic-Visual Alignment: By explicitly linking the layers of a hierarchical text decoder to the layers of a multi-resolution GAN, the latent space becomes highly interpretable. One can trace how top-layer topics like "water/sky" influence initial low-res generation, while bottom-layer topics like "red beak" refine the final high-res output.
  • Mutual Benefit: The VHE's objective forces the latent space to cover all data modes, preventing GAN mode collapse. Conversely, the GAN's adversarial loss helps the VHE learn a more accurate representation of the data distribution.

Results and Performance

The VHE-raster-scan-GAN achieved state-of-the-art Inception Scores (IS) across several benchmarks: 4.41 on CUB, 3.72 on Oxford-102 Flower, and 27.16 on MS COCO. Qualitatively, it successfully rendered specific textual details that competing models like StackGAN++ and AttnGAN often missed.

Beyond generation, the model demonstrated exceptional bidirectional flexibility. Due to the shared latent space, the same model successfully performed image-to-text generation (captioning), text-based image retrieval, and smooth latent space interpolations. It also outperformed existing methods in text-based zero-shot learning (ZSL), achieving a Top-1 accuracy of 16.7% on CUB-Hard.

Conclusion

VHE-GAN provides a robust framework for bidirectional image-text modeling. While highly effective, future work may involve incorporating explicit spatial constraints (like bounding boxes) to further improve performance on complex datasets like COCO and fine-tuning the architecture for domain-specific tasks such as facial attribute synthesis.

Summary updated

Abstract

For bidirectional joint image-text modeling, we develop variational hetero-encoder (VHE) randomized generative adversarial network (GAN), a versatile deep generative model that integrates a probabilistic text decoder, probabilistic image encoder, and GAN into a coherent end-to-end multi-modality learning framework. VHE randomized GAN (VHE-GAN) encodes an image to decode its associated text, and feeds the variational posterior as the source of randomness into the GAN image generator. We plug three off-the-shelf modules, including a deep topic model, a ladder-structured image encoder, and StackGAN++, into VHE-GAN, which already achieves competitive performance. This further motivates the development of VHE-raster-scan-GAN that generates photo-realistic images in not only a multi-scale low-to-high-resolution manner, but also a hierarchical-semantic coarse-to-fine fashion. By capturing and relating hierarchical semantic and visual concepts with end-to-end training, VHE-raster-scan-GAN achieves state-of-the-art performance in a wide variety of image-text multi-modality learning and generation tasks.
BibTeX
@inproceedings{zhang2020variational,
  author = {Hao Zhang and Bo Chen and Long Tian and Zhengjue Wang and Mingyuan Zhou},
  title = {{Variational Hetero-Encoder Randomized Generative Adversarial Networks for Joint Image-Text Modeling}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
}
Figure