Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Models

Yuge Shi, Brooks Paige, Philip Torr, Siddharth N

TL;DR

Training multimodal generative models usually requires a massive amount of perfectly paired data (like an image and its exact caption), which is expensive to produce. This paper introduces a contrastive framework that allows models to learn not just from what makes data "related," but also from what makes it "unrelated." The result is a massive boost in data efficiency: the authors show that their method can achieve the same performance as standard models using only 20% of the paired data.

The Problem: The High Cost of Paired Data

Deep Generative Models (DGMs) are highly effective at learning "multimodal" concepts—for example, understanding that the word "bird" and a picture of a bird represent the same underlying idea. However, these models are notoriously data-hungry. To train them, researchers typically need large datasets where every image is strictly paired with a corresponding text description or audio clip.

In the real world, "related" data is scarce and expensive to annotate, while "unrelated" data (random images and random text) is abundant. Most existing models simply ignore these unrelated pairs. The authors of this paper ask a fundamental question: Can we use the distinction between related and unrelated samples to help the model learn faster and better?

A Novel Contrastive Framework

The authors propose a novel contrastive framework for Multimodal Variational Autoencoders (VAEs) that explicitly models the relationship between data pairs.

Graphical model comparing standard inference to the proposed contrastive framework with a relatedness variable.
  1. The "Relatedness" Variable ($r$): As shown in the graphical model above, they introduce a Bernoulli random variable ($r$) that explicitly dictates whether two observations ($x$ and $y$) are related ($r=1$) or unrelated ($r=0$).
  2. Contrastive Loss ($L_C$): Instead of just maximizing the likelihood of related pairs, they use a "triplet-style" loss. This encourages the model to maximize the Pointwise Mutual Information (PMI) between related pairs while minimizing it for unrelated pairs. Essentially, it tells the model: "The information shared between these two should be high, and the information shared between these random two should be low."
  3. Balancing Act ($\gamma$): They found that simply using a contrastive loss can lead the model to "cheat" by generating noise to satisfy the math. To fix this, they introduced a hyperparameter $\gamma$ to upweight the standard reconstruction term, ensuring the model still produces high-quality, recognizable outputs.
  4. Estimators: Since the true "likelihood" in these models is hard to calculate, they tested two different ways to estimate it: IWAE (Importance Weighted Autoencoder) and CUBO ($\chi$ upper bound). They found that IWAE was generally more stable and effective.

Datasets and Evaluation

The framework was evaluated on two primary benchmarks to test its ability to find common concepts across different modalities:

Examples of related and unrelated pairs in the MNIST-SVHN dataset.
  • MNIST-SVHN: A challenging dataset where handwritten digits (MNIST) are paired with house numbers (SVHN) of the same value. As illustrated above, the model learns from both matching pairs (e.g., a handwritten '4' with a street sign '4') and mismatched pairs.
  • CUB Image-Captions: A complex, real-world dataset featuring 11,788 photos of birds paired with descriptive captions. This is a much harder "vision-language" task where the content of the image and text can vary significantly even when they describe the same bird.

Key Insights and Results

  • Negative Space Matters: The most important insight is that "unrelated" data isn't just noise—it provides a boundary. By seeing what a pair isn't supposed to look like, the model gains a much sharper understanding of what a correct pair is.
  • The 20% Rule: The authors demonstrated that their contrastive model trained on only 20% of a dataset could match or beat a standard model trained on 100% of the same dataset. On the MNIST-SVHN task, the contrastive MMVAE (cI-MMVAE) using 20% of the data achieved a joint coherence of ~42.7%, nearly matching the baseline's performance on 100% of the data (~42.3%).
  • The Model as a Matchmaker: Because the model learns to distinguish related from unrelated data, it can be used for label propagation. You can give the model a small amount of paired data and a mountain of unpaired data, and it can accurately "find" the correct pairs itself.
  • Performance Boost: Across various VAE architectures (like MMVAE, MVAE, and JMVAE), adding the contrastive framework consistently improved "cross-coherence"—the ability of the model to generate a correct image given a text prompt, and vice-versa.

Limitations and Future Work

While highly effective, the framework does have limitations. The results are sensitive to the value of $\gamma$; if it is too low, the model produces blurry images, but if it is too high, the benefits of the contrastive loss disappear. Additionally, while the IWAE estimator worked well, the CUBO estimator suffered from numerical stability issues on complex datasets. Finally, while the paper provides a mathematical generalization for more than two modalities, the experimental results focused primarily on pairs, leaving tri-modal (or higher) applications for future research.

Summary updated

Abstract

Multimodal learning for generative models often refers to the learning of abstract concepts from the commonality of information in multiple modalities, such as vision and language. While it has proven effective for learning generalisable representations, the training of such models often requires a large amount of related multimodal data that shares commonality, which can be expensive to come by. To mitigate this, we develop a novel contrastive framework for generative model learning, allowing us to train the model not just by the commonality between modalities, but by the distinction between "related" and "unrelated" multimodal data. We show in experiments that our method enables data-efficient multimodal learning on challenging datasets for various multimodal VAE models. We also show that under our proposed framework, the generative model can accurately identify related samples from unrelated ones, making it possible to make use of the plentiful unlabeled, unpaired multimodal data.
BibTeX
@inproceedings{shi2021relating,
  author = {Yuge Shi and Brooks Paige and Philip Torr and Siddharth N},
  title = {{Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=vhKe9UFbrJo},
}
Figure