Abstract
Traditional Latent Space Optimization (LSO) in continuous generative models often suffers from poor image quality and an inability to extrapolate beyond the training distribution. This paper presents a framework that performs LSO in the discrete latent space of a Vector Quantized Variational Autoencoder (VQ-VAE). By modeling the relationship between discrete tokens and target properties using tree-based ensembles and solving for the optimal configuration via Mixed-Integer Optimization (MIO), the authors demonstrate the ability to synthesize high-quality, out-of-distribution images—such as intensely smiling faces—even when the model was trained on a dataset restricted to neutral expressions.
Introduction
In deep generative modeling, Latent Space Optimization (LSO) aims to find samples that maximize a specific property, such as the "smileyness" of a face or the binding affinity of a molecule. Historically, this has been performed in continuous latent spaces using standard Variational Autoencoders (VAEs). However, continuous LSO faces two major hurdles: the "Extrapolation Gap," where models struggle to imagine features unseen in training, and poor image quality, often resulting in blurry or distorted artifacts.
To address these issues, this research moves optimization into a discrete space, leveraging the high-fidelity reconstruction capabilities of VQ-VAEs and the global search power of mathematical programming.
Methodology: Discrete Latent Space Optimization
The proposed pipeline treats image synthesis as a global optimization problem. The process consists of several integrated stages:
- VQ-VAE Foundation: Unlike standard VAEs, the VQ-VAE represents images as a grid of discrete tokens. This discretization helps avoid "posterior collapse" and produces sharper images.
- Tree-Based Surrogates: To map the relationship between latent tokens and the target objective (e.g., a smile), the researchers train a Gradient Boosted Regression Tree ensemble. Trees are naturally suited for the categorical nature of discrete latent variables.
- Mixed-Integer Optimization (MIO): The tree ensemble is converted into a set of linear constraints. A global solver then identifies the exact combination of tokens predicted to maximize the objective.
- Weighted Retraining: To maintain realism, the model uses an iterative loop. High-scoring generated samples are added back to the training set with higher weights, fine-tuning the VQ-VAE to better understand the desired "out-of-distribution" features.
Experimental Setup
The framework was evaluated using the CelebA-Dialog dataset. To test the model's ability to extrapolate, the training set was restricted to images with a "smiling degree" of 2 or less (neutral to slight smiles). The goal was to synthesize faces with degrees 3, 4, or 5—expressions the model had never explicitly seen.
Key Results
Quantitative Performance
The discrete LSO method significantly outperformed continuous baselines in both optimization and image quality:
- Optimization Success: The proposed method achieved a Top50 smile score of ~2.50, whereas the continuous VAE baseline failed to reach even 1.70, struggling to match the best points already present in the training data.
- Image Realism: The Fréchet Inception Distance (FID), a standard measure of image quality, improved from 50.51 to 41.69, indicating more realistic generations.
Qualitative Fidelity
Visually, the difference is stark. While continuous models often produce "ghostly" or distorted faces when pushed to extrapolate, the discrete method generates clear, realistic smiles with fewer artifacts.
Conclusion and Future Work
By combining VQ-VAEs with tree-based MIO, this research provides a principled way to perform global optimization in latent spaces. While the current study focuses on facial expressions, the authors suggest the framework is highly applicable to structured domains like chemical design and neural architecture search (NAS). Future work will focus on scaling the MIO solver to handle higher-resolution latent grids and exploring more complex, multi-objective optimization problems.
Summary updated