Abstract
This paper introduces MMEDA-II, a novel unsupervised framework designed for joint representation learning from heterogeneous data sources, specifically high-dimensional tensors (images) and matrices (text). By integrating convolutional autoencoders with matrix factorization techniques, MMEDA-II preserves spatial hierarchies in visual data while effectively fusing it with textual features. Experimental results on the Goodreads dataset demonstrate that MMEDA-II outperforms traditional supervised baselines and collective matrix factorization in downstream tasks such as genre classification and rating prediction.
Introduction
In real-world applications, data is rarely unimodal. For instance, medical diagnostics often rely on a combination of X-rays (images), clinical notes (text), and vitals (tabular data). Multimodal Representation Learning aims to compress these diverse inputs into a single, meaningful mathematical vector. However, current approaches face two primary challenges: the "label bottleneck," where supervised learning requires extensive manual labeling, and "structural limitations," where traditional methods like Collective Matrix Factorization (CMF) require flattening images into 1D vectors, thereby losing critical spatial information.
Methodology
The proposed Multi-Modal Encoder Decoder Architecture (MMEDA) addresses these gaps by employing specialized neural network components for different data modalities. For image data (tensors), a Convolutional Autoencoder (ConvAE) is used to maintain spatial structure. Textual data (matrices), processed via Word2Vec, is handled by standard feed-forward autoencoders.
The core innovation lies in the "end-to-end" fusion of these architectures. Encoded features from both modalities are concatenated and passed through a shared feed-forward network. MMEDA-II improves upon the initial MMEDA-I design by incorporating an additional matrix multiplication step to better align learned representations with input attributes. The model is trained by minimizing a composite reconstruction loss, forcing the central representation to capture the essence of all input modalities simultaneously.
Experimental Setup
The framework was evaluated using a subset of 5,000 records from the Goodreads dataset, featuring book cover images and descriptions. The learned representations were tested on two downstream tasks:
- Genre Classification: A binary classification task (Fiction vs. Non-Fiction) using a Random Forest classifier.
- Rating Prediction: A regression task to predict average user ratings using Linear Regression.
Results
MMEDA-II demonstrated superior performance across both tasks compared to several baselines, including supervised Multi-Layer Perceptrons (MLP) and CMF with various backbones like ResNet and VGG.
As shown in the results, MMEDA-II achieved the highest classification accuracy (0.70) and the lowest Mean Squared Error (0.1737) for rating prediction. These results suggest that unsupervised learning can effectively capture latent structural patterns that supervised models might overlook when focused on specific, potentially noisy labels.
Conclusion and Future Work
MMEDA-II provides a robust framework for unsupervised multimodal learning that preserves data structure and reduces the need for manual feature engineering. Future research will explore the integration of Transformer-based text encoders (e.g., BERT or GPT) to yield richer representations and extending the architecture to handle sequential data such as video.
Summary updated