Abstract
Cross-modal representation learning is an essential part of representation learning, which aims to learn latent semantic representations for modalities including texts, audio, images, videos, etc. In this chapter, we first introduce typical cross-modal representation models. After that, we review several real-world applications related to cross-modal representation learning including image captioning, visual relation detection, and visual question answering.
Introduction
Humans naturally integrate information from multiple senses; for instance, we use the visual movement of a speaker's lips to help disambiguate the sounds we hear. Computers, however, struggle with this integration. The core challenge is the semantic gap: raw pixels and text strings have entirely different structures. Cross-modal representation learning addresses this by learning "latent semantic representations" that are modality-independent. This is crucial for tasks like Zero-Shot Recognition and Cross-Media Retrieval.
Architectural Paradigms
The field has evolved through several key architectural paradigms designed to bridge the gap between vision and language:
- Joint Embedding Spaces: Models like DeViSE project image features (from CNNs) and word features (from Skip-gram) into a shared space using a hinge ranking loss. This ensures that an image of a dog is closer to the word "dog" than the word "car."
- Encoder-Decoder Frameworks: Standard for image captioning, where a CNN "encodes" an image into a vector and an RNN (like an LSTM) "decodes" that vector into a natural language sentence.
- Attention Mechanisms: Visual Attention allows decoders to focus on specific image regions when generating specific words, such as looking at the sky when generating the word "cloud."
As shown in the figure above, modern architectures often combine textual sequences with visual features extracted via Convolutional Neural Networks (CNNs). These features are fused into a higher-level representation that can then be used to classify actions or relationships, such as determining if ducks are "swimming," "walking," or "sitting."
Key Insights and Applications
The integration of these modalities provides several advantages:
- Visual Disambiguation of Text: Visual data reveals that words like "eat" and "stare" often co-occur, helping models capture implicit semantic relatedness that text alone might miss.
- Language Priors: In visual relationship detection (e.g., "person-ride-horse"), visual data alone is often sparse for rare interactions. Pre-trained word embeddings provide a "prior" that helps the model predict likely relationships even with few visual examples.
- Reasoning via Scene Graphs: The shift toward "Scene Graph Generation" represents a move toward symbolic AI, where images are converted into graphs of objects (nodes) and their relationships (edges).
Conclusion and Future Work
While significant progress has been made, several frontiers remain:
- Fine-grained Grounding: Linking specific words to specific small objects or attributes remains difficult.
- Cross-modal Reasoning: Moving beyond pattern matching to true "commonsense reasoning"—understanding why an action is occurring.
- Unsupervised Learning: Leveraging the vast amount of unlabelled multi-modal data available on the web to reduce reliance on expensive human-annotated datasets.
Summary updated