TL;DR
This paper introduces Retrieval-IVAE (RIVAE), a novel approach for learning disentangled latent factors from paired cross-modal data (like images and text). By removing the need for complex decoders and leveraging an "implicit decoder" with Jacobian regularization, RIVAE effectively identifies shared underlying factors, significantly outperforming existing methods in cross-modal retrieval tasks.
What Problem Does This Paper Address?
Learning meaningful, disentangled latent representations from complex, high-dimensional data is a long-standing challenge in machine learning. Traditional methods like Variational Autoencoders (VAEs) often struggle with training accurate decoders for such data, which can hinder their ability to identify true underlying factors. In cross-modal retrieval, where the goal is to find corresponding items across different modalities (e.g., an image and its description), this problem is exacerbated. While paired data offers a form of supervision, existing methods either rely on computationally expensive techniques or fail to guarantee the identification of true latent factors. This paper aims to address these limitations by proposing a method that can learn disentangled latent factors from paired data without requiring a complex, potentially suboptimally trained, data decoder.
Methods
The core of the proposed method, Retrieval-IVAE (RIVAE), is built upon the Identifiable VAE (IVAE) framework. The key innovations are:
- Implicit Decoder: Instead of learning a direct mapping from latent variables back to the original data (a decoder), RIVAE uses an "implicit decoder." This is achieved by inverting the learned embedding function of one modality using Jacobian regularization. This bypasses the difficult task of training a decoder for high-dimensional data.
- Identifiability through Conditioning: RIVAE modifies the IVAE by conditioning the latent variables on the query modality's embedding. This, under certain regularity conditions, allows for provable identifiability of the model's true parameters and, consequently, the disentangled latent factors.
- Cross-Modal Retrieval Loss: A specialized loss function is introduced to encourage accurate retrieval. This loss implicitly inverts the embedding function without needing a synthesis model, leveraging the learned latent space.
- Jacobian Regularization: To ensure the embedding function is well-behaved for inversion, a regularization term is applied to its Jacobian, encouraging consistent volume changes.
Datasets
The authors evaluate their approach on several datasets:
- Synth: A synthetic dataset designed to have known shared and private factors, allowing for quantitative evaluation of factor identification.
- Sprites: A dataset where sprite shapes are used to create paired data, with sprite positions and scale as shared factors.
- Split-MNIST: A dataset derived from MNIST by splitting images into left and right halves, with digit class and writing style as shared factors.
- Recipe1M: A large-scale dataset of food images and corresponding recipes, used to evaluate the model's ability to learn meaningful food-related factors like savoriness and wateriness.
Key Insights and Ideas
- The "Implicit Decoder" is a Game-Changer: The most significant insight is that by removing the explicit decoder and relying on implicit inversion via Jacobian regularization, the model can avoid a major source of failure in VAEs for complex data. This allows for more robust learning of latent factors.
- Identifiability is Key for Disentanglement: By building on the IVAE framework and conditioning on the query modality, the authors demonstrate a path to provably identifiable latent representations, which is crucial for achieving true disentanglement.
- Cross-Modal Retrieval as a Proxy for Factor Learning: The retrieval task itself, when framed correctly with the proposed loss, acts as a powerful signal for learning disentangled latent factors, even when ground-truth factor labels are not fully available.
- Learned Factors Align with Human Perception: On the Recipe1M dataset, the learned latent factors remarkably correspond to intuitive food characteristics (savoriness, wateriness, greenness), suggesting the model captures semantically meaningful variations.
Key Results
- Superior Retrieval Performance: RIVAE consistently outperforms baseline methods like Cos-Sim-LVM, Bi-VAE, DCCA, and RBi-VAE across various datasets in terms of retrieval metrics (R@1, R@5, R@10, Med-R). For instance, on the Synth dataset, RIVAE achieves an R@1 of 0.35, significantly higher than Cos-Sim-LVM (0.02).
- High Disentanglement Scores: Quantitative metrics (Disentanglement, Completeness, Informativeness) show RIVAE achieving significantly higher scores than competing methods. On the Synth dataset, RIVAE scores 0.9186 for Disentanglement, compared to 0.0017 for DCCA.
- Qualitative Evidence of Disentanglement: Latent traversal visualizations on Synth and Sprites datasets clearly demonstrate that RIVAE's latent variables correspond to distinct underlying factors, with minimal entanglement, unlike other methods.
- Meaningful Food Factors: On Recipe1M, latent traversal reveals factors that align with human perception of food characteristics, such as wateriness, savoriness, and greenness, with ingredient word clouds providing strong qualitative support.
Limitations and Future Work
- Regularization is Crucial: The ablation study highlights the critical role of the embedder regularization. Without it, performance degrades significantly or the model fails to converge, indicating the importance of this component for stable learning.
- Future Work: The authors suggest extending their approach to retrieval tasks involving more than two modalities, such as video, text, and audio data.
Summary updated