Abstract
We present a novel, alternative framework for learning generative models with goal-conditioned reinforcement learning. We define two agents, a goal conditioned agent (GC-agent) and a supervised agent (S-agent). Given a user-input initial state, the GC-agent learns to reconstruct the training set. In this context, elements in the training set are the goals. During training, the S-agent learns to imitate the GC-agent while remaining agnostic of the goals. At inference we generate new samples with the S-agent. Following a similar route as in variational auto-encoders, we derive an upper bound on the negative log-likelihood that consists of a reconstruction term and a divergence between the GC-agent policy and the (goal-agnostic) S-agent policy. We empirically demonstrate that our method is able to generate diverse and high quality samples in the task of image synthesis.
Introduction
Generative modeling is currently characterized by a trade-off between speed and fidelity. Variational Auto-Encoders (VAEs) are computationally efficient but often produce blurry results, while Diffusion Models (DMs) generate high-fidelity images but require hundreds or thousands of iterative steps. This paper introduces a novel framework that treats generative modeling as a navigation problem within a Reinforcement Learning (RL) environment. By framing the generation process as an agent learning to "walk" toward a specific data point, the authors achieve high-quality synthesis with significantly improved inference efficiency.
Methodology: Generative Modeling as Navigation
The proposed Variational Agent (V-agent) framework utilizes Goal-Conditioned Reinforcement Learning (GCRL) to bridge the gap between VAEs and DMs. The system consists of two distinct components:
- The GC-agent (Goal-Conditioned): This agent acts as an expert during training. Given a specific image from the training set as its "goal," it learns a policy to reach that goal from a fixed starting state by selecting from a set of proposals (potential image modifications).
- The S-agent (Supervised): This is the generative model used at inference. It learns to imitate the GC-agent's trajectories without knowing the specific target goal. By learning the general "flow" of the data manifold, the S-agent can generate new samples independently.
Technical innovations include a surrogate loss function—an upper bound on the negative log-likelihood similar to the ELBO in VAEs—and a discrete action space that prevents mode collapse. Crucially, while DMs often use horizons of $H=1000$, this framework is effective with a horizon of only $H=16$.
Experimental Results
The researchers evaluated the framework on the MNIST and Fashion-MNIST benchmarks, comparing it against standard VAEs and Diffusion Models.
Key Findings:
- Reconstruction Quality: The GCRL approach achieved lower Mean Squared Error (MSE) and better Frechet Inception Distance (FID) for reconstructions than VAEs. For instance, on Fashion-MNIST, the model reached an MSE of 0.10 compared to the VAE's 0.13.
- Step Efficiency: The model produced diverse, high-quality images in just 16 steps, whereas the DM baseline required 1000 steps to achieve its results.
- Generation Quality: While Diffusion Models still maintain an edge in pure FID scores for new samples (e.g., 1.94 vs 19.46 on MNIST), the V-agent provides a significantly faster alternative that outperforms VAEs in visual quality.
Conclusion and Future Work
By framing generative modeling as a sequence of decisions in a Markov Decision Process, this work opens new avenues for applying RL techniques to image synthesis. While current experiments were limited to $28 \times 28$ grayscale images, the authors suggest that scaling to high-resolution color images and incorporating techniques like Hindsight Experience Replay (HER) could further bridge the performance gap with state-of-the-art Diffusion Models.
Summary updated