Abstract: Cross-modal representation learning is an essential part of representation learning, which aims to learn semantic representations for different modalities including text, audio, image and video, etc., and their connections. In this chapter, we introduce the development of cross-modal representation learning from shallow to deep, and from respective to unified in terms of model architectures and learning mechanisms for different modalities and tasks. After that, we review how cross-modal capabilities can contribute to complex real-world applications.
Introduction
Humans naturally experience the world through multiple modalities—sight, sound, and language—integrating them seamlessly to form a coherent understanding. In contrast, artificial intelligence has historically struggled to bridge these disparate data streams. The core challenge lies in the heterogeneity of the data. Text is a high-density, abstract signal where a single word carries significant semantic weight. Images and audio, however, are low-density raw signals; a single pixel or a millisecond of sound is virtually meaningless in isolation. Furthermore, text is inherently sequential, while images are spatial.
Cross-modal representation learning addresses these challenges by creating a "common semantic space." In this shared mathematical environment, related concepts from different modalities—such as the word "apple" and a photograph of the fruit—are mapped to proximal locations, allowing machines to perform complex reasoning across diverse inputs.
The Evolution of Cross-Modal Architectures
The development of cross-modal learning has progressed through three distinct historical phases, moving from simple feature concatenation to unified foundation models.
Shallow Learning
Early models focused on augmenting standard text algorithms, such as Word2Vec, with visual context. By using Convolutional Neural Networks (CNNs) to extract global image features and concatenating them with word vectors, researchers enabled models to disambiguate polysemous words based on surrounding imagery. For example, visual cues can help a model distinguish whether the word "Apple" refers to the fruit or the technology corporation.
Deep Specialized Learning
As deep learning matured, task-specific architectures emerged to handle complex interactions. Attention mechanisms became central to tasks like Visual Question Answering (VQA), allowing models to dynamically focus on specific image regions relevant to a textual query. Another significant development was the use of scene graphs, which represent images as structured networks of objects (nodes) and their relationships (edges), aligning more closely with the compositional and logical structure of human language.
Deep Unified Pre-training
The current frontier involves unified Transformer-based architectures. Unlike previous specialized models, Transformers can process both text tokens and image patches (via Vision Transformers) using the same underlying mechanism.
- Single-Stream Models: These feed text and images into a single encoder for deep, early-stage fusion.
- Two-Stream Models (e.g., CLIP): These encode text and images separately into a shared space, prioritizing retrieval efficiency and enabling "open-vocabulary" recognition of objects never explicitly seen during supervised training.
- Prompt Learning: Techniques like Colorful Prompt Tuning (CPT) reformulate complex downstream tasks into simple completion problems, facilitating zero-shot or few-shot learning by bridging the gap between pre-training and specific applications.
Benchmarks and Datasets
Progress in the field is driven by a wide array of benchmarks. Reasoning and VQA tasks are evaluated on datasets like GQA, VQA-CP, and COCO-QA, while image captioning and retrieval rely on MS-COCO, Flickr30k, and Visual Genome. The shift toward unified models has been fueled by web-scale pre-training datasets, such as the 400 million image-text pairs used to train OpenAI’s CLIP.
Key Insights and Future Directions
A critical insight from recent research is the impact of information density on pre-training. Image models (like MAE) require much higher masking rates—often exceeding 75%—compared to the 15% typically used in text models. This is because images are highly redundant; the model must be forced to work harder to learn high-level semantics rather than simple local correlations.
We are currently transitioning from specialized "vision" or "language" models toward general-purpose "foundation models." However, challenges remain. Patch-based methods can struggle with precise spatial localization compared to object-based methods. Additionally, the computational cost of training these models is immense. Future work is increasingly focused on "embodied AI," where models interact with the physical world, and the expansion of multimodal computation into 3D environments and the metaverse.
Summary updated