Abstract
Latent Space Bayesian Optimization (LSBO) combines generative models, typically Variational Autoencoders (VAE), with Bayesian Optimization (BO) to generate de-novo objects of interest. However, LSBO faces challenges due to the mismatch between the objectives of BO and VAE, resulting in poor exploration capabilities. In this paper, we propose novel contributions to enhance LSBO efficiency and overcome this challenge. We first introduce the concept of latent consistency/inconsistency as a crucial problem in LSBO, arising from the VAE-BO mismatch. To address this, we propose the Latent Consistent Aware-Acquisition Function (LCA-AF) that leverages consistent points in LSBO. Additionally, we present LCA-VAE, a novel VAE method that creates a latent space with increased consistent points through data augmentation in latent space and penalization of latent inconsistencies. Combining LCA-VAE and LCA-AF, we develop LCA-LSBO. Our approach achieves high sample-efficiency and effective exploration, emphasizing the significance of addressing latent consistency through the novel incorporation of data augmentation in latent space within LCA-VAE in LSBO. We showcase the performance of our proposal via de-novo image generation and de-novo chemical design tasks.
Introduction: The Latent Inconsistency Problem
Bayesian Optimization (BO) is a powerful tool for optimizing expensive black-box functions. However, when the search space is high-dimensional (e.g., images or molecules), standard BO struggles. Latent Space Bayesian Optimization (LSBO) addresses this by training a Variational Autoencoder (VAE) to map high-dimensional data into a low-dimensional latent space. Optimization then occurs in this latent space, where a surrogate model guides the search.
A critical issue in LSBO is the "latent inconsistency" that arises from the mismatch between VAE and BO objectives. VAEs are optimized to reconstruct existing data points. BO, conversely, seeks to explore sparse regions where data is absent. In these regions, the mapping is often unreliable: a point $z$ decoded to $x$ and then re-encoded might map to a distant point $z'$. This inconsistency breaks the surrogate model's ability to predict the properties of generated objects accurately, leading to "off-target" generation.
The LCA-LSBO Framework
To resolve this, we introduce LCA-LSBO (Latent Consistency Aware LSBO), which consists of two primary components:
1. LCA-AF: Latent Consistent Aware-Acquisition Function
Instead of standard acquisition functions that only consider objective value and uncertainty, LCA-AF prioritizes "consistent" points. It employs a "successive cycles" approach to identify latent points that remain stable after multiple decode-encode iterations, ensuring the BO evaluates points that the generative model can reliably reconstruct.
2. LCA-VAE: Enhancing Latent Consistency
To provide the BO with a reliable search space, LCA-VAE introduces a Latent Consistency Loss (LCL). This loss penalizes the distance between a latent point $z$ and its reconstruction $z^1 = E(D(z))$. Furthermore, we employ Latent Data Augmentation. By sampling "fake" latent points from sparse regions and forcing the VAE to make them consistent, we effectively "pave" the latent space ahead of the BO's exploration. During the BO phase, this augmentation is dynamically moved toward the most promising regions.
Experimental Results
We evaluated LCA-LSBO across several challenging tasks:
- Zero-Shot Image Generation: On MNIST and Fashion MNIST, the model was tasked with generating a class it had never seen during training (e.g., generating '0' after seeing only '1'-'9'). While baseline LSBO methods failed, LCA-LSBO successfully generated recognizable target objects within a few iterations.
- Chemical Design (ZINC250K): In optimizing docking scores for the KAT1 protein, LCA-LSBO outperformed the best molecules in the original dataset, achieving a score of -9.58 (compared to the dataset best of -8.80) in only 500 iterations.
Conclusion
LCA-LSBO addresses the fundamental mismatch in latent space optimization by explicitly modeling and enforcing latent consistency. By "paving" the latent space through targeted data augmentation and consistency-aware acquisition, our method enables efficient exploration in high-dimensional spaces, making it a robust candidate for complex design tasks in science and engineering where data is scarce and exploration is critical.
Summary updated