Bridged Variational Autoencoders for Joint Modeling of Images and Attributes

Ravindra Yadav, Ashish Sardana, Vinay P. Namboodiri, Rajesh M. Hegde

Introduction

Generative modeling has seen significant success in capturing single-modality data distributions. However, real-world data is inherently multimodal, often consisting of images paired with descriptive attributes. Learning the correspondence between these modalities is challenging due to their disparate statistical properties. This paper introduces Bridged Variational Autoencoders (BVAE), a novel architecture designed to achieve state-of-the-art cross-modal generation while maintaining high parameter efficiency.

Problem Statement

Existing multimodal VAE approaches generally fall into three categories, each with distinct drawbacks:

  1. Joint Models: Effective when all modalities are present but fail during inference if a modality is missing.
  2. Retrofitted Models: Require separate encoders for every possible combination of inputs, leading to a "parameter explosion" as the number of modalities increases.
  3. Product-of-Experts (PoE): Parameter-efficient but often struggle to train individual encoders effectively, resulting in poor performance when only a single modality is available.

Proposed Method: Bridged Variational Autoencoders

The BVAE addresses these limitations by introducing a Bridge Encoder ($Enc_{BR}$). This component allows the model to "inherit" knowledge from a pre-trained joint model rather than training entirely new networks for missing data scenarios.

Comparison of multimodal VAE architectures: (a) Joint, (b) Retrofit, (c) Product-of-experts, and (d) the proposed Bridge Model.

The architecture utilizes unimodal encoders and a central bridge containing small, fully connected networks ($FC_{x \to y}$ and $FC_{y \to x}$). These networks are designed to "hallucinate" the latent features of a missing modality based on the available input. During training, the model maximizes joint log-likelihood. When a modality is absent at inference, the bridge ensures the latent space remains consistent with the joint representation, allowing the use of the optimized joint decoder. This approach is highly scalable; while retrofit models require $2^n - 1$ encoders, BVAE only needs $2 \times \binom{n}{2}$ bridge connections.

Experimental Setup

The BVAE was evaluated on three primary benchmarks:

  • CelebA: A face attributes dataset using 18 visually distinctive attributes (e.g., Bald, Mustache, Smiling).
  • MNIST: Handwritten digits where labels serve as the attribute modality.
  • Trimodal Dataset: A synthetic combination of MNIST, Fashion-MNIST, and shared tags to test scaling capabilities.

Results and Discussion

The BVAE demonstrates significant improvements in both efficiency and generative quality.

Efficiency and Likelihood

The model achieved a 40% reduction in parameters for bimodal data and a 70% reduction for trimodal data compared to state-of-the-art retrofit models. On MNIST, the BVAE reached a test marginal log-likelihood of -98.76, nearly matching the "ideal" joint model (-98.93) and outperforming MVAE (-100.52).

Qualitative Performance

Unlike previous methods like Bi-VCCA, which often produce blurry "mean" images, the BVAE maintains stochasticity in its latent space. This allows for the generation of diverse and distinct images for the same attribute set.

Qualitative results on the CelebA dataset showing image generation across varying attributes and samples.

In attribute recognition tasks on CelebA, BVAE outperformed TELBO, JMVAE, and MVAE in 11 out of 18 categories. Visual inspections confirm that BVAE generates clearer images with fewer artifacts compared to competitors like JMVAE, which often collapse to "average" faces.

Conclusion

Bridged Variational Autoencoders provide a robust framework for multimodal joint modeling. By leveraging "information inheritance" and latent space hallucination, the BVAE achieves superior cross-modal generation and recognition performance with a fraction of the parameters required by traditional models. Future work will explore the application of this bridging mechanism to high-dimensional unstructured text and natural trimodal data.

Summary updated

Abstract

Generative models have recently shown the ability to realistically generate data and model the distribution accurately. However, joint modeling of an image with the attribute that it is labeled with requires learning a cross modal correspondence between image and attribute data. Though the information present in a set of images and its attributes possesses completely different statistical properties altogether, there exists an inherent correspondence that is challenging to capture. Various models have aimed at capturing this correspondence either through joint modeling of a variational autoencoder or through separate encoder networks that are then concatenated. We present an alternative by proposing a bridged variational autoencoder that allows for learning cross-modal correspondence by incorporating cross-modal hallucination losses in the latent space. In comparison to the existing methods, we have found that by using a bridge connection in latent space we not only obtain better generation results, but also obtain highly parameter-efficient model which provide 40% reduction in training parameters for bimodal dataset and nearly 70% reduction for trimodal dataset. We validate the proposed method through comparison with state of the art methods and benchmarking on standard datasets.
BibTeX
@inproceedings{yadav2020bridged,
  author = {Ravindra Yadav and Ashish Sardana and Vinay P. Namboodiri and Rajesh M. Hegde},
  title = {{Bridged Variational Autoencoders for Joint Modeling of Images and Attributes}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093565},
}
Figure