Cross-modal Variational Alignment of Latent Spaces

Thomas Theodoridis, Theocharis Chatzis, Vassilios Solachidis, Kosmas Dimitropoulos, Petros DarasWorkshops @ CVPR 2020 · DOI

Abstract

Cross-modal learning seeks to relate disparate data types, such as images and text, by mapping them into a shared latent space. This paper introduces a novel framework that aligns these spaces using a specialized "variational alignment" branch. By modeling an intermediate distribution between two Variational Autoencoders (VAEs) and employing Wasserstein distance for distribution matching, the method creates a more effective translation mechanism. Experimental results in food ingredient recognition and 3D hand pose estimation show that this approach significantly outperforms current state-of-the-art models.

Introduction

A central challenge in cross-modal machine learning is the inherent structural difference between data types. For example, mapping a grid of pixels (an image) to a set of discrete text labels (ingredients) requires a representation that captures the nuances of both simultaneously. Existing methods often struggle with this alignment, relying on basic generative models or discriminative mappings that fail to bridge the "language" gap between modalities effectively. This paper addresses these limitations by introducing a three-branch architecture designed to find common ground between distinct data distributions.

Proposed Methodology

The authors propose a three-branch architecture centered on Variational Autoencoders (VAEs) to facilitate precise cross-modal translation.

The proposed three-branch architecture featuring the Variational Alignment "Mapper" branch.

As illustrated in the architecture diagram, the system comprises:

  1. The Upper Branch ($VAE_1$): A multimodal network that learns to map the source modality ($M_1$, e.g., an image) directly to the target modality ($M_2$, e.g., ingredients).
  2. The Lower Branch ($VAE_2$): A single-modal network that auto-encodes the target modality ($M_2 \to M_2$), establishing a "gold standard" latent space for that specific data type.
  3. The Variational Alignment Branch (The "Mapper"): Instead of a direct projection, this branch inserts an additional VAE-like structure that projects the latent space of the first modality onto the second through an intermediate distribution ($LS_m$).

To ensure robust alignment, the framework utilizes the Wasserstein distance as a loss function ($L^W$). Unlike standard KL-divergence, the Wasserstein distance provides a smoother measure of the transformation required between probability distributions, leading to tighter latent space integration. The training process is executed in three distinct phases: independent training of the VAE branches, training of the mapper while other components are frozen, and finally, end-to-end fine-tuning.

Experimental Setup

The framework was evaluated on four datasets across two distinct domains to prove its versatility:

  • Food Image Analysis: The researchers used Yummly-28K and Recipe1M to test ingredient recognition. For Recipe1M, a subset of approximately 300,000 recipes was utilized for training and testing.
  • 3D Hand Pose Estimation: The Rendered Handpose Dataset (RHD) and Stereo Hand Pose Tracking Benchmark (STB) were used to predict 3D keypoints from 2D images, representing a regression-based challenge.

Results and Discussion

The proposed method consistently outperformed previous state-of-the-art (SOTA) models. In ingredient recognition on Recipe1M, the model achieved an F1 score of 49.18, surpassing specialized transformer architectures. In hand pose estimation on RHD, it achieved a Mean End-Point-Error (EPE) of 15.61, effectively halving the error of previous leading methods (which scored 30.42).

t-SNE visualization demonstrating the successful alignment of image and ingredient features in the latent space.

Qualitative alignment was verified using t-SNE visualizations, which showed that the "mapped" image features clustered much more closely with actual ingredient features than raw features did. This confirms the mapper's ability to act as an effective bridge between modalities. Furthermore, the architecture proved effective for both classification and regression tasks, suggesting that the variational alignment mechanism is a fundamental improvement for latent space matching regardless of the output type.

Conclusion

The use of an intermediate "mapper" distribution and Wasserstein-based alignment allows for superior cross-modal translation while preserving the unique characteristics of each modality. While currently limited to pairs of modalities, future work will explore extending this framework to handle multi-modal data (e.g., image, audio, and text simultaneously) and streamlining the complex phased training process.

Summary updated

BibTeX
@inproceedings{theodoridis2020cross,
  author = {Thomas Theodoridis and Theocharis Chatzis and Vassilios Solachidis and Kosmas Dimitropoulos and Petros Daras},
  title = {{Cross-modal Variational Alignment of Latent Spaces}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  doi = {10.1109/cvprw50498.2020.00488},
}
Figure