Happy People - Image Synthesis as Black-Box Optimization Problem in the Discrete Latent Space of Deep Generative Models

Steffen Jung, Jan Christian Schwedhelm, Claudia Schillings, Margret KeuperarXiv 2023 · DOI · arXiv

Abstract

Traditional Latent Space Optimization (LSO) in continuous generative models often suffers from poor image quality and an inability to extrapolate beyond the training distribution. This paper presents a framework that performs LSO in the discrete latent space of a Vector Quantized Variational Autoencoder (VQ-VAE). By modeling the relationship between discrete tokens and target properties using tree-based ensembles and solving for the optimal configuration via Mixed-Integer Optimization (MIO), the authors demonstrate the ability to synthesize high-quality, out-of-distribution images—such as intensely smiling faces—even when the model was trained on a dataset restricted to neutral expressions.

Introduction

In deep generative modeling, Latent Space Optimization (LSO) aims to find samples that maximize a specific property, such as the "smileyness" of a face or the binding affinity of a molecule. Historically, this has been performed in continuous latent spaces using standard Variational Autoencoders (VAEs). However, continuous LSO faces two major hurdles: the "Extrapolation Gap," where models struggle to imagine features unseen in training, and poor image quality, often resulting in blurry or distorted artifacts.

To address these issues, this research moves optimization into a discrete space, leveraging the high-fidelity reconstruction capabilities of VQ-VAEs and the global search power of mathematical programming.

Methodology: Discrete Latent Space Optimization

The proposed pipeline treats image synthesis as a global optimization problem. The process consists of several integrated stages:

  1. VQ-VAE Foundation: Unlike standard VAEs, the VQ-VAE represents images as a grid of discrete tokens. This discretization helps avoid "posterior collapse" and produces sharper images.
  2. Tree-Based Surrogates: To map the relationship between latent tokens and the target objective (e.g., a smile), the researchers train a Gradient Boosted Regression Tree ensemble. Trees are naturally suited for the categorical nature of discrete latent variables.
  3. Mixed-Integer Optimization (MIO): The tree ensemble is converted into a set of linear constraints. A global solver then identifies the exact combination of tokens predicted to maximize the objective.
  4. Weighted Retraining: To maintain realism, the model uses an iterative loop. High-scoring generated samples are added back to the training set with higher weights, fine-tuning the VQ-VAE to better understand the desired "out-of-distribution" features.
The proposed framework combining VQ-VAE retraining with global optimization in discrete latent space.

Experimental Setup

The framework was evaluated using the CelebA-Dialog dataset. To test the model's ability to extrapolate, the training set was restricted to images with a "smiling degree" of 2 or less (neutral to slight smiles). The goal was to synthesize faces with degrees 3, 4, or 5—expressions the model had never explicitly seen.

Key Results

Quantitative Performance

The discrete LSO method significantly outperformed continuous baselines in both optimization and image quality:

  • Optimization Success: The proposed method achieved a Top50 smile score of ~2.50, whereas the continuous VAE baseline failed to reach even 1.70, struggling to match the best points already present in the training data.
  • Image Realism: The Fréchet Inception Distance (FID), a standard measure of image quality, improved from 50.51 to 41.69, indicating more realistic generations.

Qualitative Fidelity

Visually, the difference is stark. While continuous models often produce "ghostly" or distorted faces when pushed to extrapolate, the discrete method generates clear, realistic smiles with fewer artifacts.

Qualitative comparison of generated faces. The bottom row (red) shows the proposed discrete LSO results, exhibiting clearer features and more intense smiles than the continuous VAE results in the middle row.

Conclusion and Future Work

By combining VQ-VAEs with tree-based MIO, this research provides a principled way to perform global optimization in latent spaces. While the current study focuses on facial expressions, the authors suggest the framework is highly applicable to structured domains like chemical design and neural architecture search (NAS). Future work will focus on scaling the MIO solver to handle higher-resolution latent grids and exploring more complex, multi-objective optimization problems.

Summary updated

Abstract

In recent years, optimization in the learned latent space of deep generative models has been successfully applied to black-box optimization problems such as drug design, image generation or neural architecture search. Existing models thereby leverage the ability of neural models to learn the data distribution from a limited amount of samples such that new samples from the distribution can be drawn. In this work, we propose a novel image generative approach that optimizes the generated sample with respect to a continuously quantifiable property. While we anticipate absolutely no practically meaningful application for the proposed framework, it is theoretically principled and allows to quickly propose samples at the mere boundary of the training data distribution. Specifically, we propose to use tree-based ensemble models as mathematical programs over the discrete latent space of vector quantized VAEs, which can be globally solved. Subsequent weighted retraining on these queries allows to induce a distribution shift. In lack of a practically relevant problem, we consider a visually appealing application: the generation of happily smiling faces (where the training distribution only contains less happy people) - and show the principled behavior of our approach in terms of improved FID and higher smile degree over baseline approaches.
BibTeX
@misc{jung2023happy,
  author = {Steffen Jung and Jan Christian Schwedhelm and Claudia Schillings and Margret Keuper},
  title = {{Happy People - Image Synthesis as Black-Box Optimization Problem in the Discrete Latent Space of Deep Generative Models}},
  year = {2023},
  eprint = {2306.06684},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.06684},
}
Figure