Multimodal Representation Learning With Text and Images

Aishwarya Jayagopal, Ankireddy Monica Aiswarya, Ankita Garg, Srinivasan Kolumam NandakumararXiv 2022 · DOI · arXiv

Abstract

This paper introduces MMEDA-II, a novel unsupervised framework designed for joint representation learning from heterogeneous data sources, specifically high-dimensional tensors (images) and matrices (text). By integrating convolutional autoencoders with matrix factorization techniques, MMEDA-II preserves spatial hierarchies in visual data while effectively fusing it with textual features. Experimental results on the Goodreads dataset demonstrate that MMEDA-II outperforms traditional supervised baselines and collective matrix factorization in downstream tasks such as genre classification and rating prediction.

Introduction

In real-world applications, data is rarely unimodal. For instance, medical diagnostics often rely on a combination of X-rays (images), clinical notes (text), and vitals (tabular data). Multimodal Representation Learning aims to compress these diverse inputs into a single, meaningful mathematical vector. However, current approaches face two primary challenges: the "label bottleneck," where supervised learning requires extensive manual labeling, and "structural limitations," where traditional methods like Collective Matrix Factorization (CMF) require flattening images into 1D vectors, thereby losing critical spatial information.

Methodology

The proposed Multi-Modal Encoder Decoder Architecture (MMEDA) addresses these gaps by employing specialized neural network components for different data modalities. For image data (tensors), a Convolutional Autoencoder (ConvAE) is used to maintain spatial structure. Textual data (matrices), processed via Word2Vec, is handled by standard feed-forward autoencoders.

Architecture of the MMEDA-II model.

The core innovation lies in the "end-to-end" fusion of these architectures. Encoded features from both modalities are concatenated and passed through a shared feed-forward network. MMEDA-II improves upon the initial MMEDA-I design by incorporating an additional matrix multiplication step to better align learned representations with input attributes. The model is trained by minimizing a composite reconstruction loss, forcing the central representation to capture the essence of all input modalities simultaneously.

Experimental Setup

The framework was evaluated using a subset of 5,000 records from the Goodreads dataset, featuring book cover images and descriptions. The learned representations were tested on two downstream tasks:

  1. Genre Classification: A binary classification task (Fiction vs. Non-Fiction) using a Random Forest classifier.
  2. Rating Prediction: A regression task to predict average user ratings using Linear Regression.

Results

MMEDA-II demonstrated superior performance across both tasks compared to several baselines, including supervised Multi-Layer Perceptrons (MLP) and CMF with various backbones like ResNet and VGG.

Performance comparison of MMEDA-II against baseline architectures.

As shown in the results, MMEDA-II achieved the highest classification accuracy (0.70) and the lowest Mean Squared Error (0.1737) for rating prediction. These results suggest that unsupervised learning can effectively capture latent structural patterns that supervised models might overlook when focused on specific, potentially noisy labels.

Conclusion and Future Work

MMEDA-II provides a robust framework for unsupervised multimodal learning that preserves data structure and reduces the need for manual feature engineering. Future research will explore the integration of Transformer-based text encoders (e.g., BERT or GPT) to yield richer representations and extending the architecture to handle sequential data such as video.

Summary updated

Abstract

In recent years, multimodal AI has seen an upward trend as researchers are integrating data of different types such as text, images, speech into modelling to get the best results. This project leverages multimodal AI and matrix factorization techniques for representation learning, on text and image data simultaneously, thereby employing the widely used techniques of Natural Language Processing (NLP) and Computer Vision. The learnt representations are evaluated using downstream classification and regression tasks. The methodology adopted can be extended beyond the scope of this project as it uses Auto-Encoders for unsupervised representation learning.
BibTeX
@misc{jayagopal2022multimodal,
  author = {Aishwarya Jayagopal and Ankireddy Monica Aiswarya and Ankita Garg and Srinivasan Kolumam Nandakumar},
  title = {{Multimodal Representation Learning With Text and Images}},
  year = {2022},
  eprint = {2205.00142},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.00142},
}
Figure