Abstract
We present a conditional variational auto-encoder (VAE) which, to avoid the substantial cost of training from scratch, uses an architecture and training objective capable of leveraging a foundation model in the form of a pretrained unconditional VAE. To train the conditional VAE, we only need to train an artifact to perform amortized inference over the unconditional VAE's latent variables given a conditioning input. We demonstrate our approach on tasks including image inpainting, for which it outperforms state-of-the-art GAN-based approaches at faithfully representing the inherent uncertainty. We conclude by describing a possible application of our inpainting model, in which it is used to perform Bayesian experimental design for the purpose of guiding a sensor.
Introduction
Training state-of-the-art generative models on high-resolution images is computationally prohibitive, often requiring an entire GPU-year for a top-tier Variational Auto-Encoder (VAE). While Generative Adversarial Networks (GANs) offer faster training times, they suffer from "mode dropping," where the model ignores valid data possibilities to focus on a few high-probability modes.
In critical domains like medical imaging or sensor guidance, capturing the full diversity of the data distribution—the "mass-covering" property—is essential. Missing a rare but valid mode, such as a tumor in an X-ray, can lead to catastrophic failures. This work introduces Inference in a Pretrained Artifact (IPA), a framework that bridges the gap between the training efficiency of GANs and the robust probabilistic coverage of VAEs by repurposing pretrained unconditional models.
The IPA Framework
The core innovation of IPA is the transformation of a pretrained unconditional VAE into a conditional one without retraining the entire architecture. The method leverages a "foundation model" and introduces a specialized training regime.
Methodology
- Decoder Reuse: The decoder of the pretrained unconditional VAE is kept frozen. This ensures the model retains the high-quality generative capabilities of the original foundation model.
- Partial Encoder: Instead of a full encoder, a smaller "partial encoder" is introduced. This component takes only the conditioning input $y$ (e.g., a masked image) and maps it to the latent space of the pretrained model.
- The Objective: The partial encoder is trained to produce a distribution over latent variables $z$ that matches the posterior distribution the original "full" encoder would have produced had it seen the complete, unmasked image $x$.
- Preventing Posterior Collapse: Because the fixed decoder cannot see the conditioning input directly, the model is forced to utilize the latent variables to represent missing information, effectively bypassing the common VAE issue of posterior collapse.
Theoretical analysis confirms that if the original VAE is well-trained, the IPA objective is equivalent to minimizing the distance to the true posterior distribution, providing a rigorous foundation for the amortized inference approach.
Experimental Results
The authors evaluated IPA across standard vision benchmarks (CIFAR-10, ImageNet, FFHQ) and specialized tasks like Edges2Shoes and medical imaging.
Efficiency and Diversity
IPA demonstrates significant gains in training speed. For the FFHQ-256 dataset, IPA required under 7 GPU-weeks, a massive reduction from the 1 GPU-year needed for the original unconditional model.
Crucially, IPA maintains superior diversity compared to GANs. On the LPIPS-GT metric, which measures how well samples cover the ground truth distribution, IPA consistently outperformed state-of-the-art models like CoModGAN. This confirms that the model successfully avoids mode-dropping and captures the inherent uncertainty of image completion tasks.
Bayesian Optimal Experimental Design (BOED)
The utility of IPA's uncertainty representation was tested in a medical context using the NIH Chest X-ray 14 dataset. The model was tasked with deciding where a sensor should focus next to diagnose an illness. The IPA-based system achieved high diagnostic accuracy (AUROC) significantly faster than baselines, demonstrating its value in high-stakes, sequential decision-making environments.
Discussion and Limitations
While IPA offers a powerful path toward efficient conditional generation, it is inherently limited by the quality of the pretrained unconditional VAE. Any artifacts or biases in the foundation model, such as a lack of bilateral symmetry, are inherited by the conditional model. Furthermore, the theoretical justification assumes that latent variables capture all structural information, which may be less effective for tasks like super-resolution where fine-grained pixel details are highly dependent on the input. Future work may explore more expressive partial encoders, such as normalizing flows, to further refine the fit to the posterior distribution.
Summary updated