Introduction
Generative modeling has seen significant success in capturing single-modality data distributions. However, real-world data is inherently multimodal, often consisting of images paired with descriptive attributes. Learning the correspondence between these modalities is challenging due to their disparate statistical properties. This paper introduces Bridged Variational Autoencoders (BVAE), a novel architecture designed to achieve state-of-the-art cross-modal generation while maintaining high parameter efficiency.
Problem Statement
Existing multimodal VAE approaches generally fall into three categories, each with distinct drawbacks:
- Joint Models: Effective when all modalities are present but fail during inference if a modality is missing.
- Retrofitted Models: Require separate encoders for every possible combination of inputs, leading to a "parameter explosion" as the number of modalities increases.
- Product-of-Experts (PoE): Parameter-efficient but often struggle to train individual encoders effectively, resulting in poor performance when only a single modality is available.
Proposed Method: Bridged Variational Autoencoders
The BVAE addresses these limitations by introducing a Bridge Encoder ($Enc_{BR}$). This component allows the model to "inherit" knowledge from a pre-trained joint model rather than training entirely new networks for missing data scenarios.
The architecture utilizes unimodal encoders and a central bridge containing small, fully connected networks ($FC_{x \to y}$ and $FC_{y \to x}$). These networks are designed to "hallucinate" the latent features of a missing modality based on the available input. During training, the model maximizes joint log-likelihood. When a modality is absent at inference, the bridge ensures the latent space remains consistent with the joint representation, allowing the use of the optimized joint decoder. This approach is highly scalable; while retrofit models require $2^n - 1$ encoders, BVAE only needs $2 \times \binom{n}{2}$ bridge connections.
Experimental Setup
The BVAE was evaluated on three primary benchmarks:
- CelebA: A face attributes dataset using 18 visually distinctive attributes (e.g., Bald, Mustache, Smiling).
- MNIST: Handwritten digits where labels serve as the attribute modality.
- Trimodal Dataset: A synthetic combination of MNIST, Fashion-MNIST, and shared tags to test scaling capabilities.
Results and Discussion
The BVAE demonstrates significant improvements in both efficiency and generative quality.
Efficiency and Likelihood
The model achieved a 40% reduction in parameters for bimodal data and a 70% reduction for trimodal data compared to state-of-the-art retrofit models. On MNIST, the BVAE reached a test marginal log-likelihood of -98.76, nearly matching the "ideal" joint model (-98.93) and outperforming MVAE (-100.52).
Qualitative Performance
Unlike previous methods like Bi-VCCA, which often produce blurry "mean" images, the BVAE maintains stochasticity in its latent space. This allows for the generation of diverse and distinct images for the same attribute set.
In attribute recognition tasks on CelebA, BVAE outperformed TELBO, JMVAE, and MVAE in 11 out of 18 categories. Visual inspections confirm that BVAE generates clearer images with fewer artifacts compared to competitors like JMVAE, which often collapse to "average" faces.
Conclusion
Bridged Variational Autoencoders provide a robust framework for multimodal joint modeling. By leveraging "information inheritance" and latent space hallucination, the BVAE achieves superior cross-modal generation and recognition performance with a fraction of the parameters required by traditional models. Future work will explore the application of this bridging mechanism to high-dimensional unstructured text and natural trimodal data.
Summary updated