Multimodal Generative Models for Scalable Weakly-Supervised Learning

Mike Wu, Noah D. Goodman

Introduction

In the real world, data is inherently multimodal—an event might be described by an image, a text caption, and audio simultaneously. Learning a joint representation across these modalities is a powerful goal for generative modeling. However, multimodal data is notoriously expensive to collect. While we often have access to massive amounts of unpaired, single-modality data, we typically only possess a small set of fully paired examples. This scenario, known as weak supervision, presents a significant challenge for existing models.

Previous generative models, such as the Joint Multimodal VAE (JMVAE), struggle to scale beyond two modalities. To handle missing data at test time, these models often require a separate inference network for every possible combination of modalities. For $N$ modalities, this leads to an exponential explosion of $2^N$ inference networks, making the approach intractable for complex real-world datasets. Furthermore, existing models frequently rely on multi-stage training pipelines that are difficult to optimize.

Methodology: The Multimodal Variational Autoencoder (MVAE)

The authors introduce the Multimodal Variational Autoencoder (MVAE), which addresses the scaling problem through two primary innovations: a Product-of-Experts (PoE) architecture and a sub-sampled training paradigm.

Product-of-Experts (PoE) Inference Network

The core architectural breakthrough of the MVAE is the assumption that modalities are conditionally independent given a shared latent variable $z$. Mathematically, this implies that the true joint posterior is proportional to the product of individual unimodal posteriors. Since the product of Gaussian distributions is itself a Gaussian, the MVAE only needs to learn $N$ unimodal inference networks. To handle any combination of missing data, the model simply multiplies the outputs of the available "expert" networks together.

The MVAE architecture: (a) the underlying graphical model, (b) the Product-of-Experts (PoE) inference network, and (c) the mechanism for handling missing modalities.

Sub-sampled Training Paradigm

A common pitfall in multimodal training is that if a model is only trained on complete data, the individual unimodal networks never learn to operate independently. Conversely, training them entirely independently fails to capture the joint distribution. The MVAE employs a sub-sampled objective: for every gradient step, the model optimizes the Evidence Lower Bound (ELBO) using the full joint data, the individual unimodal data, and $k$ random subsets of the modalities. This single-stage training forces the model to learn robust representations regardless of which data components are missing.

Experimental Evaluation

The authors evaluated the MVAE across a diverse range of datasets, including standard image/label sets (MNIST, FashionMNIST), complex multi-digit canvases (MultiMNIST), and high-dimensional face images with 18 visual attributes (CelebA). They also tested the model on computer vision transformations and machine translation tasks (English-Vietnamese).

Key Insights

  • Linear Scaling: By leveraging the PoE factorization, the parameter count scales linearly ($O(N)$) rather than exponentially ($O(2^N)$), representing a massive architectural simplification.
  • Statistical Strength: When treating the 18 CelebA attributes as independent modalities rather than a single vector, the model performed better, suggesting that joint training allows the model to share statistical strength across modalities.
  • Bootstrapping: The MVAE excels at using massive amounts of unpaired data to learn marginal distributions, requiring only a tiny fraction of paired data to learn the bridge between modalities.

Results and Discussion

The MVAE demonstrates state-of-the-art performance while maintaining high parameter efficiency. On bi-modal tasks, it matches the JMVAE's performance with roughly half the parameters.

Accuracy vs. Log Number of Examples on the MultiMNIST dataset, comparing VAE, JMVAE, and MVAE. The MVAE shows superior performance and efficiency in weakly-supervised settings.

In terms of weak supervision, the MVAE matched the accuracy of a fully supervised neural network on FashionMNIST using two orders of magnitude fewer paired examples. It also successfully handled extreme scaling, managing 19 distinct modalities in the CelebA experiments and achieving near-maximum translation performance on low-resource language pairs using only 5% of aligned sentence pairs.

Limitations and Future Work

Despite its successes, the MVAE inherits the characteristic blurriness of standard VAE generative backbones. Additionally, while the weak-supervision results for translation are promising, the absolute performance does not yet compete with fully supervised state-of-the-art models. Future work may involve integrating the MVAE paradigm with modern architectures like Transformers and attention mechanisms to improve generative quality and translation accuracy.

Summary updated

BibTeX
@inproceedings{wu2018multimodal,
  author = {Mike Wu and Noah D. Goodman},
  title = {{Multimodal Generative Models for Scalable Weakly-Supervised Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=rJbuaLW_ZB},
}
Figure