Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Space

Yangming Li, Chieh-Hsin Lai, Carola-Bibiane Schönlieb, Yuki Mitsufuji, Stefano Ermon

Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Space

Distributional Reinforcement Learning (RL) seeks to model the full return distribution rather than just the expected value. While Deep Generative Models (DGMs) excel at complex density estimation, their inherent nonlinearity creates a mismatch with the linear Bellman equation. Bellman Diffusion addresses this by modeling the gradient and scalar fields of a distribution directly, preserving linearity and enabling efficient integration with RL frameworks.

The Nonlinearity Bottleneck

Modern DGMs like Energy-Based Models (EBMs) or score-matching diffusion models typically operate on log-densities or normalized scores. However, the Bellman equation—the cornerstone of temporal difference learning—is a linear operator. Applying nonlinear DGMs to recursive RL updates often requires expensive trajectory sampling or discrete approximations (e.g., C51). Bellman Diffusion overcomes this "nonlinearity bottleneck" by shifting the modeling target to fields that respect linear operations.

Methodology: Field Modeling and Sampling

Bellman Diffusion models the probability density $p(x)$ through its constituent fields: the scalar field $s_\varphi(x) \approx p(x)$ and the gradient field $g_\phi(x) \approx \nabla p(x)$. Since both the identity and gradient operators are linear, the Bellman update can be applied directly to these neural proxies.

Visualization of the scalar and gradient fields learned by Bellman Diffusion on a 2D mixture of Gaussians.

To sample from the learned distribution, the authors propose a novel Stochastic Differential Equation (SDE): $$dx(t) = \nabla p_{target}(x(t)) dt + \sqrt{p_{target}(x(t))} dw(t)$$ Unlike standard Langevin dynamics, which uses the score function $\nabla \log p(x) = \frac{\nabla p(x)}{p(x)}$, this SDE avoids division by the density. This prevents numerical instability in low-density regions and allows the model to accurately capture unbalanced modes. To scale to high dimensions, the authors employ a "slice trick," projecting high-dimensional densities onto 1D random directions for efficient training.

Experimental Results

The framework was evaluated on synthetic benchmarks, UCI tabular datasets, and RL environments such as Frozen Lake and Cart Pole.

Bellman Diffusion successfully captures discontinuous distributions, such as these 1D disjoint uniform spans.

A key finding is the model's ability to handle discontinuous distributions. As shown in the figure above, Bellman Diffusion accurately learns uniform distributions over disjoint spans by capturing the sharp derivatives at boundaries—a task typically difficult for continuous neural networks. In RL tasks, Bellman Diffusion achieved 1.5x faster convergence than the widely-used C51 histogram baseline in the Cart Pole environment, demonstrating superior stability and efficiency.

Conclusion and Future Work

Bellman Diffusion provides a theoretically grounded bridge between generative modeling and RL. By prioritizing linearity, it enables high-fidelity return distribution estimation that integrates seamlessly with the Bellman equation. While currently sensitive to certain hyperparameters like the variance term ($\epsilon$), future work aims to scale the model to high-resolution image domains and develop discrete-time convergence theorems to further bridge the gap between theory and practice.

Summary updated

Abstract

Deep Generative Models (DGMs), including Score-based Generative Models, have made significant progress in approximating complex continuous distributions. However, their application to Markov Decision Processes (MDPs), particularly in distributional Reinforcement Learning (RL), is underexplored. The field remains dominated by classical histogram-based methods, which suffer from discretization errors, leading to instability and slower convergence. This work highlights that this gap stems from the nonlinear operators used in modern DGM's modelings, which map neural network functions to the target distribution. These nonlinearities conflict with the linearity required by the Bellman equation, which relates the return distribution of a state to a linear combination of future states' return distributions. To address this, we introduce Bellman Diffusion, a new DGM that preserves the necessary linearity by modeling both the gradient and scalar fields. We propose a novel divergence-based training technique to optimize neural network proxies and introduce a new stochastic differential equation for sampling. With these innovations, Bellman Diffusion is guaranteed to converge to the target distribution. Our experiments show that Bellman Diffusion not only achieves accurate field estimations and serves as an effective image generator, but also converges $1.5\times$ faster than traditional histogram-based baselines in distributional RL tasks. This work paves the way for the effective integration of DGMs into MDP applications, enabling more advanced decision-making frameworks.
BibTeX
@misc{li2024bellman,
  author = {Yangming Li and Chieh-Hsin Lai and Carola-Bibiane Schönlieb and Yuki Mitsufuji and Stefano Ermon},
  title = {{Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Space}},
  year = {2024},
  eprint = {2410.01796},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.01796},
}
Figure