Learning Generative Models with Goal-conditioned Reinforcement Learning

Mariana Vargas Vieyra, Pierre MénardarXiv 2023 · OpenReview

Abstract

We present a novel, alternative framework for learning generative models with goal-conditioned reinforcement learning. We define two agents, a goal conditioned agent (GC-agent) and a supervised agent (S-agent). Given a user-input initial state, the GC-agent learns to reconstruct the training set. In this context, elements in the training set are the goals. During training, the S-agent learns to imitate the GC-agent while remaining agnostic of the goals. At inference we generate new samples with the S-agent. Following a similar route as in variational auto-encoders, we derive an upper bound on the negative log-likelihood that consists of a reconstruction term and a divergence between the GC-agent policy and the (goal-agnostic) S-agent policy. We empirically demonstrate that our method is able to generate diverse and high quality samples in the task of image synthesis.

Introduction

Generative modeling is currently characterized by a trade-off between speed and fidelity. Variational Auto-Encoders (VAEs) are computationally efficient but often produce blurry results, while Diffusion Models (DMs) generate high-fidelity images but require hundreds or thousands of iterative steps. This paper introduces a novel framework that treats generative modeling as a navigation problem within a Reinforcement Learning (RL) environment. By framing the generation process as an agent learning to "walk" toward a specific data point, the authors achieve high-quality synthesis with significantly improved inference efficiency.

Methodology: Generative Modeling as Navigation

The proposed Variational Agent (V-agent) framework utilizes Goal-Conditioned Reinforcement Learning (GCRL) to bridge the gap between VAEs and DMs. The system consists of two distinct components:

  1. The GC-agent (Goal-Conditioned): This agent acts as an expert during training. Given a specific image from the training set as its "goal," it learns a policy to reach that goal from a fixed starting state by selecting from a set of proposals (potential image modifications).
  2. The S-agent (Supervised): This is the generative model used at inference. It learns to imitate the GC-agent's trajectories without knowing the specific target goal. By learning the general "flow" of the data manifold, the S-agent can generate new samples independently.
Overview of the proposed framework where a GC-agent selects proposals to reach a goal state, and an S-agent learns to imitate these trajectories.

Technical innovations include a surrogate loss function—an upper bound on the negative log-likelihood similar to the ELBO in VAEs—and a discrete action space that prevents mode collapse. Crucially, while DMs often use horizons of $H=1000$, this framework is effective with a horizon of only $H=16$.

Experimental Results

The researchers evaluated the framework on the MNIST and Fashion-MNIST benchmarks, comparing it against standard VAEs and Diffusion Models.

Comparison of MSE and FID scores across MNIST and Fashion-MNIST datasets.

Key Findings:

  • Reconstruction Quality: The GCRL approach achieved lower Mean Squared Error (MSE) and better Frechet Inception Distance (FID) for reconstructions than VAEs. For instance, on Fashion-MNIST, the model reached an MSE of 0.10 compared to the VAE's 0.13.
  • Step Efficiency: The model produced diverse, high-quality images in just 16 steps, whereas the DM baseline required 1000 steps to achieve its results.
  • Generation Quality: While Diffusion Models still maintain an edge in pure FID scores for new samples (e.g., 1.94 vs 19.46 on MNIST), the V-agent provides a significantly faster alternative that outperforms VAEs in visual quality.

Conclusion and Future Work

By framing generative modeling as a sequence of decisions in a Markov Decision Process, this work opens new avenues for applying RL techniques to image synthesis. While current experiments were limited to $28 \times 28$ grayscale images, the authors suggest that scaling to high-resolution color images and incorporating techniques like Hindsight Experience Replay (HER) could further bridge the performance gap with state-of-the-art Diffusion Models.

Summary updated

Abstract

We present a novel, alternative framework for learning generative models with goal-conditioned reinforcement learning. We define two agents, a goal conditioned agent (GC-agent) and a supervised agent (S-agent). Given a user-input initial state, the GC-agent learns to reconstruct the training set. In this context, elements in the training set are the goals. During training, the S-agent learns to imitate the GC-agent while remaining agnostic of the goals. At inference we generate new samples with the S-agent. Following a similar route as in variational auto-encoders, we derive an upper bound on the negative log-likelihood that consists of a reconstruction term and a divergence between the GC-agent policy and the (goal-agnostic) S-agent policy. We empirically demonstrate that our method is able to generate diverse and high quality samples in the task of image synthesis.
BibTeX
@misc{vieyra2023learning,
  author = {Mariana Vargas Vieyra and Pierre Ménard},
  title = {{Learning Generative Models with Goal-conditioned Reinforcement Learning}},
  year = {2023},
  eprint = {2303.14811},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.14811},
}
Figure