Multimodal Generative Models for Compositional Representation Learning

Mike Wu, Noah D. GoodmanarXiv 2019 · arXiv

Introduction

In human cognition, understanding is inherently multimodal. We perceive objects through a synthesis of visual, tactile, and linguistic inputs. However, in machine learning, models are frequently specialized for single modalities. While multimodal models exist, they often rely on massive, perfectly paired datasets or utilize "ad-hoc" objective functions that fail to correctly bound the joint probability of the data. This often results in models that either ignore cross-modal relationships or fail to capture the nuances of individual modalities.

Researchers at Stanford have proposed a principled mathematical framework to address these challenges. Their approach balances learning individual data types with learning the relationships between them, demonstrating that training vision models alongside language forces the emergence of "compositional" representations—abstract understandings of scenes as discrete objects with specific properties and relations.

Methodology: The $\mathcal{L}_M$ Objective

The core of this work is a new family of objectives derived from variational divergences. Unlike previous models that might focus solely on the joint distribution $p(x, y)$, the authors introduce the $\mathcal{L}_M$ objective, which decomposes the training process into four distinct terms:

  1. Marginal Likelihoods: Two terms ensure the model can accurately reconstruct an image or text independently.
  2. Conditional Likelihoods: Two terms focus on cross-modal "translation," ensuring the model can generate text from an image and vice versa.

This decomposition ensures a mathematically sound bound on the joint marginal likelihood, correcting biases found in earlier models like MVAE and JMVAE. By correctly bounding the likelihood, the resulting "VAEVAE" model achieves a superior balance between reconstruction quality and cross-modal translation.

Model Agnosticism and Scalability

The framework is designed to be model-agnostic, extending beyond Variational Autoencoders (VAEs) to include Generative Adversarial Networks (VAEGANs) and Flow-based models (VAENVPs). This flexibility allows researchers to select the optimal architecture for specific modalities—for instance, using GANs for continuous image data and VAEs for discrete text. To handle the stability issues common in multimodal GANs, the authors implement a 1-Lipschitz constraint on the discriminator, inspired by Wasserstein GANs.

To ensure the model scales as more modalities (e.g., sound or touch) are added, the framework utilizes a Product-of-Experts (PoE) approach. This allows for efficient information combination from multiple sensors without a linear explosion in complexity.

Language as a Scaffold for Compositionality

A significant insight of this research is the role of language as a "compositional prior." Because language is structured around discrete entities and relational predicates (e.g., "a red square above a blue circle"), aligning visual features with linguistic ones forces the vision model to move beyond pixel-level correlations. Instead, the model is incentivized to learn object-level representations.

This hypothesis was tested using the ShapeWorld benchmark and a Tree Reconstruction Error (TRE) metric. The results indicated that multimodal features were significantly more compositional than their unimodal counterparts. Specifically, a VAEGAN trained with language was nearly twice as effective at capturing object-oriented structures compared to a standard GAN.

Experimental Results

The framework was evaluated across diverse datasets, including MNIST, CelebA, Chairs in Context (CIC), and MS-COCO.

  • Sample Quality: The VAEGAN variant achieved superior image quality, with a Fréchet Inception Distance (FID) of 0.40 on MNIST, compared to 1.31 for standard VAEs.
  • Weak Supervision: The model demonstrated robustness to sparse data, learning high-quality representations even when only 1% of the data was paired. The remaining 99% of unpaired data was used to learn the general structure of each modality.
  • Downstream Utility: When learned multimodal features were integrated into a YOLOv3 object detector, the detector reached high accuracy significantly faster than with standard features. Furthermore, adding VAEGAN features to a visual relation predictor (LSVRD) improved predicate prediction (e.g., "sitting on") by approximately 10% without requiring explicit language labels during the downstream task.

Limitations and Future Work

Despite these advancements, challenges remain. VAE-based components still exhibit some blurriness in complex datasets like MS-COCO, and GAN components require meticulous hyperparameter tuning to avoid mode collapse. While the theory supports $K$ modalities, most experiments were bimodal. Future research will focus on scaling to four or more simultaneous modalities and extending the framework to video to capture temporal compositionality.

Summary updated

Abstract

As deep neural networks become more adept at traditional tasks, many of the most exciting new challenges concern multimodality---observations that combine diverse types, such as image and text. In this paper, we introduce a family of multimodal deep generative models derived from variational bounds on the evidence (data marginal likelihood). As part of our derivation we find that many previous multimodal variational autoencoders used objectives that do not correctly bound the joint marginal likelihood across modalities. We further generalize our objective to work with several types of deep generative model (VAE, GAN, and flow-based), and allow use of different model types for different modalities. We benchmark our models across many image, label, and text datasets, and find that our multimodal VAEs excel with and without weak supervision. Additional improvements come from use of GAN image models with VAE language models. Finally, we investigate the effect of language on learned image representations through a variety of downstream tasks, such as compositionally, bounding box prediction, and visual relation prediction. We find evidence that these image representations are more abstract and compositional than equivalent representations learned from only visual data.
BibTeX
@misc{wu2019multimodal,
  author = {Mike Wu and Noah D. Goodman},
  title = {{Multimodal Generative Models for Compositional Representation Learning}},
  year = {2019},
  eprint = {1912.05075},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.05075},
}
Figure