Multimodal Deep Learning

Cem Akkus, Luyang Chu, Vladana Djakovic, Steffen Jauch-Walser, Philipp Koch, Giacomo Loss, Christopher Marquardt, Marco Moldovan, Nadja Sauter, Maximilian Schneider, Rickmer Schulte, Karol Urbanczyk, Jann Goschenhofer, Christian Heumann, Rasmus Hvingelby, Daniel Schalk, Matthias AßenmacherarXiv 2023 · arXiv

TL;DR

This comprehensive survey explores the rapidly evolving landscape of Multimodal Deep Learning, bridging the gap between Natural Language Processing (NLP) and Computer Vision (CV). It provides a structured overview of architectures that translate between modalities, enhance representation learning, and tackle diverse tasks—from image captioning to generative art—within unified, large-scale frameworks.

What Problem Does This Paper Address?

Human perception is inherently multimodal; we understand the world by simultaneously processing sight, sound, and language. Traditional Artificial Intelligence, however, has largely developed in "silos," with NLP and CV progressing as independent subfields.

The core challenge addressed in this work is the integration of these modalities. This involves two primary hurdles:

  1. Numerical Representation: How do we represent unstructured data like text and images in a shared mathematical space (embeddings) that captures their semantic relationships?
  2. Architecture Design: How do we build models that can effectively fuse these different data streams to perform complex tasks like "Visual Commonsense Reasoning" or generating photorealistic images from short text prompts?

Methods

The paper reviews a progression of methodologies that have defined the field over the last decade:

  • Attention and Transformers: The authors highlight the shift from Recurrent Neural Networks (RNNs) to Transformer architectures. By using self-attention, these models bypass the "bottleneck" of traditional encoder-decoders, allowing them to process sequences in parallel and capture long-distance dependencies.
  • Contrastive Learning: A pivotal technique (exemplified by models like CLIP and ALIGN) where the model is trained to align image and text embeddings in a shared latent space. Instead of predicting exact captions, the model learns to match a photo with its correct description among many distractors.
  • Generative Modeling: The survey tracks the evolution from Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) to modern Diffusion Models (like GLIDE and DALL-E 2). These models learn to "reverse" a noise process to create high-fidelity images guided by text.
  • Unified Multipurpose Models: The authors discuss "foundation models" like Gato and Flamingo. These are massive, unified architectures designed to handle hundreds of different tasks—ranging from chatting to controlling robotic arms—across multiple modalities using few-shot learning.
FIGURE 1: LMU seal (left) style-transferred to Van Gogh’s Sunflower painting (center) and blended with the prompt - Van Gogh, sunflowers - via CLIP+VGAN (right).

Datasets

The paper emphasizes that the "scaling revolution" is driven by massive, often noisy, web-scale datasets:

  • NLP: Common Crawl and "The Pile" (825 GB of diverse text).
  • CV: ImageNet (14 million images) and Objects365.
  • Multimodal: LAION-5B, currently the largest openly accessible dataset with 5.85 billion image-text pairs, and WIT (Wikipedia-based).
  • Benchmarks: The authors point to VQA (Visual Question Answering) and VCR (Visual Commonsense Reasoning) as critical for testing whether models truly "understand" the interplay between sight and language.

Key Insights and Ideas

  • The Power of Scale: A recurring theme is that increasing model parameters and data volume often leads to emergent capabilities. For example, GPT-3 (175B parameters) can perform tasks it wasn't explicitly trained for simply through "in-context learning."
  • Symbol Grounding: Pure language models often struggle with "concrete" concepts because they lack visual context. Grounding text in images helps models understand that "a house is bigger than a person," a fact often omitted in written text because it is considered "common sense."
  • Efficiency vs. Generativity: Contrastive models (like CLIP) are incredibly data-efficient for classification and retrieval but lack the flexibility of generative models. Modern SOTA approaches often combine both, using a contrastive model as a "critic" to guide a generative "artist."

Key Results

  • Image Captioning: The M2 Transformer achieved a state-of-the-art CIDEr score of 131.2 on the MS COCO dataset by using a "meshed" connectivity that exploits both low-level and high-level visual relationships.
  • Zero-Shot Robustness: CLIP demonstrated that models trained on 400M image-text pairs are far more robust to "distribution shifts" (e.g., recognizing a sketch of a banana vs. a photo) than models trained on standard labeled datasets like ImageNet.
  • Few-Shot Learning: Flamingo (80B parameters) set new benchmarks by outperforming specialized models on six different tasks while only being shown a handful of examples.

Limitations and Future Work

The authors are candid about the "growing pains" of the field:

  • Societal Bias: Models trained on internet data frequently inherit and amplify harmful stereotypes (e.g., associating "human being" more strongly with men than women).
  • Environmental and Financial Costs: Training these models is prohibitively expensive and energy-intensive. Training BERT once has a carbon footprint roughly equivalent to a trans-American flight.
  • The "Black Box" Problem: As models grow more complex, they become less interpretable. The authors call for more research into why these models make certain decisions, especially for sensitive applications like medical diagnosis.
  • Beyond 2D Images: The epilogue suggests the next frontier is video. Extending these architectures to the temporal dimension remains a significant computational and algorithmic challenge.

Summary updated

Abstract

This book is the result of a seminar in which we reviewed multimodal approaches and attempted to create a solid overview of the field, starting with the current state-of-the-art approaches in the two subfields of Deep Learning individually. Further, modeling frameworks are discussed where one modality is transformed into the other, as well as models in which one modality is utilized to enhance representation learning for the other. To conclude the second part, architectures with a focus on handling both modalities simultaneously are introduced. Finally, we also cover other modalities as well as general-purpose multi-modal models, which are able to handle different tasks on different modalities within one unified architecture. One interesting application (Generative Art) eventually caps off this booklet.
BibTeX
@misc{akkus2023multimodal,
  author = {Cem Akkus and Luyang Chu and Vladana Djakovic and Steffen Jauch-Walser and Philipp Koch and Giacomo Loss and Christopher Marquardt and Marco Moldovan and Nadja Sauter and Maximilian Schneider and Rickmer Schulte and Karol Urbanczyk and Jann Goschenhofer and Christian Heumann and Rasmus Hvingelby and Daniel Schalk and Matthias Aßenmacher},
  title = {{Multimodal Deep Learning}},
  year = {2023},
  eprint = {2301.04856},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.04856},
}
Figure