Abstract
This paper introduces a conditional generative model that links disparate data modalities—such as text, speech, and images—by mapping their embeddings into a shared, constrained latent space. By employing a "proxy variable trick" to align modality-specific latent spaces during training, the model enables independent conditional inference. Experimental results on variations of the MNIST dataset demonstrate that the proposed framework successfully generates clear, recognizable images from text or speech descriptions, even for concept combinations not encountered during training.
Introduction
Humans possess a remarkable ability to cross-reference information from different senses, such as visualizing an object upon hearing its name. For artificial intelligence, replicating this cross-modality concept learning is significantly more challenging. Existing systems often struggle to generalize to unseen combinations (e.g., a "blue number five") and frequently produce blurry, "average" images when mapping directly from one modality to another (e.g., text-to-pixels). This blurriness occurs because direct regression fails to capture the underlying semantic structure. The authors address these issues by proposing a framework that relates modality-independent concepts within a common latent manifold.
Methods
The proposed method utilizes a constrained optimization procedure involving separate auto-encoders for each modality.
- Embedding Extraction: High-dimensional data is first reduced to semantic low-dimensional embeddings, such as word2vec for text or MFCC for speech.
- The Proxy Variable Trick: To allow for independent conditional inference (e.g., generating an image from text alone), the authors assign separate latent spaces ($l_x, l_y$) to each modality.
- Constrained Mapping: During training, the objective function minimizes the distance between the probability distributions of these separate latent spaces, forcing them to align.
- Generative Inference: Once the latent spaces are synchronized, a text embedding can be passed through the text encoder to obtain a latent representation, which is then processed by the image decoder to generate a visual output.
The authors evaluated various architectures for the image component, including Multi-layered Perceptrons (MLP) and Convolutional Neural Networks (CNN), in both standard and Variational Auto-encoder (VAE) configurations.
Experimental Setup
The model was tested on several variations of the MNIST handwritten digit dataset:
- Double MNIST: Horizontal concatenations of two digits (100 possible combinations).
- Colored Double MNIST: Double digits with red, green, or blue attributes (900 possible combinations).
- Multi-modal Inputs: Text represented by word2vec embeddings (trained on Wikipedia) and speech represented by MFCC features of digit recordings.
To evaluate generalization, the authors intentionally removed specific digit and color combinations from the training set to test the model's ability to handle unseen concepts.
Results and Discussion
The VAE-based proposed model demonstrated significant improvements over baseline methods in both quality and accuracy.
- Image Quality: Qualitatively, the proposed model produced clear and recognizable digits, whereas direct mapping baselines often resulted in blurry "ghost" images. This is because the VAE-based approach learns to select a specific clear representation rather than averaging across all possible matches.
- Quantitative Performance: The convolutional VAE (conv-vae) model achieved the highest Peak Signal to Noise Ratio (PSNR). For text-to-image generation of double digits, it reached 15.82 dB, outperforming the baseline's 14.73 dB.
- Zero-Shot Generalization: Most notably, the model successfully generated images for digit and color combinations it had never seen during training. This indicates that the model learned the underlying concepts of digits and colors rather than simply memorizing training pairs.
Conclusion
This research demonstrates that mapping between low-dimensional semantic subspaces is more effective for cross-modal generation than direct pixel-space mapping. The "proxy variable trick" provides a flexible architecture for independent encoding and decoding across modalities. While currently limited to the MNIST domain, future work will focus on scaling the model to handle complex natural language captions and higher-dimensional natural images.
Summary updated