Cross-modal representation learning and generation

Huafeng Liu, Jingjing Chen, Liang Li, Bing‐Kun Bao, Zechao Li, Jiaying Liu, Liqiang NieJournal of Image and Graphics 2023 · DOI

Abstract

With the explosive growth of heterogeneous multimedia data, cross-modal learning has become a focal point for both academia and industry. This review explores the two core pillars of the field: cross-modal representation and cross-modal generation. Representation learning aims to leverage complementarity across modalities to extract effective features, while generation focuses on semantic consistency to enable data transformation. This paper analyzes recent advancements, including traditional statistical methods, Transformer-based large models, and the evolution from GANs to Diffusion models, while comparing domestic and international research landscapes.

Bridging the Semantic and Heterogeneity Gaps

The core challenge in cross-modal learning lies in bridging two fundamental gaps:

  1. The Semantic Gap: The difficulty of connecting low-level digital features (like pixel values) to high-level human concepts (like "joy" or "a sunset").
  2. The Heterogeneity Gap: The structural differences between data types, such as the grid-based nature of images versus the sequential, discrete nature of natural language.

Research in this field seeks to map these disparate formats into a shared mathematical space where semantically related items—regardless of their original modality—are positioned closely together.

Evolution of Representation Learning

The field has transitioned from traditional statistical alignment to massive-scale pre-training.

Traditional Approaches

  • Unified Representation: These methods map multiple modalities into a single, shared feature vector. A common example is the Multimodal Autoencoder, which learns a joint latent space.
  • Coordinated Representation: Instead of a single vector, modalities are kept in separate spaces but synchronized via constraints. Techniques like Canonical Correlation Analysis (CCA) or triplet losses ensure that a "dog" image and the word "dog" have high similarity scores.

The Transformer Era and Multi-modal Large Models

The advent of the Transformer architecture revolutionized the field, leading to two primary architectural paradigms:

  • Single-stream Architectures: Models like VisualBERT and UNITER feed both text and image features into a single Transformer stack from the beginning, allowing for early and deep interaction between modalities.
  • Dual-stream Architectures: Models like ViLBERT and the influential CLIP (Contrastive Language-Image Pre-training) use separate encoders for each modality. These are often aligned via contrastive learning on massive datasets to learn highly transferable visual representations.

The Generative Revolution: From GANs to Diffusion

Cross-modal generation has seen a "quantum leap" in fidelity and diversity, moving from competitive modeling to iterative refinement.

  • Generative Adversarial Networks (GANs): Early success was driven by GANs, which use a game-theoretic competition between a generator and a discriminator. While effective, they often suffer from training instability.
  • Autoregressive Models: Models like the original DALL-E treated image generation as a sequence-to-sequence task, predicting "image tokens" much like a language model predicts words.
  • Diffusion Models (DDPMs): Modern state-of-the-art systems (e.g., DALL-E 2, Stable Diffusion) utilize diffusion processes. By iteratively removing noise from a signal, these models offer superior stability and the ability to capture complex, high-resolution data distributions.

Benchmarks and Datasets

The progress of cross-modal AI is inextricably linked to the availability of large-scale datasets. While MS-COCO and Flickr30K remain foundational for image-text tasks, the field has moved toward massive-scale self-supervision using datasets like the 400M pair collection used by CLIP.

Notably, Chinese research has contributed significant domestic benchmarks, such as Wukong (100M pairs) and the "WuDao" datasets. These are essential for developing AI that understands Chinese linguistic and cultural nuances, ensuring that cross-modal models are effective in localized contexts.

Future Frontiers

Despite rapid progress, several bottlenecks define the next frontier of research:

  • Efficiency: Training large models requires immense computational power. Future research is focusing on "lightweight" architectures and efficient fine-tuning techniques like LoRA.
  • Fine-grained Control: Current models often struggle with attribute decoupling (e.g., confusing colors between two objects in a prompt). Achieving precise, pixel-level control is a major objective.
  • Interpretability: Moving beyond "black box" models to understand why specific cross-modal associations are made is critical for safety and reliability.
  • Evaluation: Traditional metrics like BLEU often fail to capture semantic nuance. Developing human-aligned evaluation frameworks is a priority for the next generation of cross-modal systems.

Summary updated

Abstract

多媒体数据持续呈现爆发式增长并显现出异源异构的特性,因此跨模态学习领域研究逐渐引起学术和工业界的关注。跨模态表征与生成是跨模态学习的两大核心基础问题。跨模态表征旨在利用多种模态之间的互补性剔除模态之间的冗余,从而获得更为有效的特征表示;跨模态生成则是基于模态之间的语义一致性,实现不同模态数据形式上的相互转换,有助于提高不同模态间的迁移能力。本文系统地分析了国际与国内近年来跨模态表征与生成领域的重要研究进展,包括传统跨模态表征学习、多模态大模型表示学习、图像到文本的跨模态转换和跨模态图像生成。其中,传统跨模态表征学习探讨了跨模态统一表征和跨模态协同表征,多模态大模型表示学习探讨了基于Transformer的模型研究,图像到文本的跨模态转换探讨了图像视频的语义描述、视频字幕语义分析和视觉问答等领域的发展,跨模态图像生成从不同模态信息的跨模态联合表示方法、图像的跨模态生成技术和基于预训练的特定域图像生成阐述了跨模态生成方面的进展。本文详细综述了上述各个子领域研究的挑战性,对比了国内外研究方面的进展情况,梳理了发展脉络和学术研究的前沿动态。最后,根据上述分析展望了跨模态表征与生成的发展趋势和突破口。
BibTeX
@article{liu2023cross,
  author = {Huafeng Liu and Jingjing Chen and Liang Li and Bing‐Kun Bao and Zechao Li and Jiaying Liu and Liqiang Nie},
  title = {{Cross-modal representation learning and generation}},
  journal = {Journal of Image and Graphics},
  year = {2023},
  doi = {10.11834/jig.230035},
}
Figure