Improved Multimodal Deep Learning with Variation of Information

Kihyuk Sohn, Wenling Shang, Honglak LeeNeurIPS 2014

Introduction

In multimodal deep learning, the primary goal is to train models that can understand and leverage the complex relationships between different types of data, such as images and their corresponding text captions. Traditionally, the standard approach to learning these joint representations relies on Maximum Likelihood (ML).

However, ML exhibits a significant weakness in multimodal contexts: it attempts to model the entire joint distribution at once, which includes the highly complex internal structures of each individual modality (the "marginal distributions"). Because intra-modality correlations (e.g., the relationship between adjacent pixels in an image) are typically much stronger than inter-modality correlations (e.g., the relationship between a specific pixel and a word), ML training often becomes "distracted." The model learns to represent each modality in isolation but fails to adequately capture the high-level semantic bridge connecting them. Consequently, these models perform poorly during "unimodal queries"—scenarios where one modality, such as the descriptive text, is missing at test time and the model must reason about it using only the available data.

To address this, researchers have introduced a novel training objective called Minimum Variation of Information (MinVI). By shifting the focus from standard Maximum Likelihood to minimizing the "information distance" between modalities, MinVI creates models that are significantly better at reasoning about missing data.

Methodology: The MinVI Objective and MDRNN

The proposed approach relies on two major conceptual shifts and a novel network architecture to fundamentally change how multimodal models are trained.

1. Minimizing Variation of Information

Instead of maximizing the joint likelihood, the authors propose minimizing the Variation of Information (VI). VI is an information-theoretic measure that quantifies the distance between two random variables. By minimizing VI, the training objective forces the model to focus specifically on conditional relationships: "What does the image tell me about the text?" and "What does the text tell me about the image?"

The core insight here is that conditional distributions are "cleaner" than marginal distributions. Marginal distributions (e.g., the distribution of all possible images) are incredibly complex and multi-modal. Conditional distributions (e.g., the distribution of text given a specific image) are much sharper and easier for a model to learn. The authors provide mathematical proof demonstrating that learning these conditional transition operators is sufficient to recover the full joint distribution of the data.

2. Multi-Prediction Training and the MDRNN

To implement the MinVI objective, the researchers utilize Multi-Prediction (MP) training. This involves constructing a recurrent encoding structure where the model iteratively attempts to predict a subset of missing input variables given the rest.

MDRNN Architecture

This concept is extended to deep architectures through the Multimodal Deep Recurrent Neural Network (MDRNN). As illustrated above, the MDRNN performs multiple "bottom-up" and "top-down" passes. If an image is provided but the text is missing, the model uses the image to generate a high-level joint representation. This representation then "hallucinates" the missing text features in a feedback loop, iteratively refining the final representation over several steps.

Experiments and Key Results

The researchers evaluated the MinVI approach and the MDRNN architecture across three distinct benchmarks: a toy task using the MNIST dataset, and two large-scale real-world datasets (MIR-Flickr and PASCAL VOC 2007).

Handling Missing Data: The Split-MNIST Task

To test the model's ability to handle missing modalities, the authors split MNIST digit images into left and right halves, treating each half as a separate modality. During testing, only the left half of the digit was provided as a query.

MNIST Reconstructions

The visual results demonstrate a stark contrast between standard ML training and the proposed MinVI approach. The MinVI (MP) model successfully reconstructs the missing right half of the digits with high fidelity, whereas the standard ML model produces blurry or incorrect hallucinations. Quantitatively, the MinVI approach reduced the error rate to 6.58%, a massive improvement over the 14.98% error rate of the standard ML-trained model.

Real-World Multimodal Datasets

The MDRNN also achieved state-of-the-art results on complex image-text datasets:

  • MIR-Flickr: On this dataset of 1 million images and user tags, the MDRNN achieved a Mean Average Precision (mAP) of 0.686 for multimodal queries and 0.607 for unimodal (image-only) queries. This significantly outperformed previous Multimodal Deep Boltzmann Machine (DBM) baselines.
  • PASCAL VOC 2007: Augmented with Flickr user tags, the method reached 81.5% mAP with multimodal queries, representing a substantial jump over the baseline of 74.5%.

Limitations and Future Directions

While MinVI and the MDRNN offer substantial improvements in multimodal representation learning, the authors note a few limitations and avenues for future work:

  1. Inference Latency: Because the MDRNN relies on a recurrent structure that requires multiple iterations (typically 10–20 in these experiments) to "fill in" missing data, it is computationally more expensive at test time compared to standard single-pass feed-forward networks.
  2. Approximate Posteriors: The current energy-based models utilize factorized distributions for efficiency, which may limit their expressiveness. Exploring deep networks that allow for exact posterior inference is a promising next step.
  3. Scalability to Continuous Text: The text features in these experiments relied on simple Bag-of-Words vectors. Adapting the MinVI objective to work seamlessly with modern sequence-based text models, such as Transformers, would be highly valuable for contemporary multimodal applications.

Summary updated

BibTeX
@inproceedings{sohn2014improved,
  author = {Kihyuk Sohn and Wenling Shang and Honglak Lee},
  title = {{Improved Multimodal Deep Learning with Variation of Information}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2014},
}
Figure