Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Space
Distributional Reinforcement Learning (RL) seeks to model the full return distribution rather than just the expected value. While Deep Generative Models (DGMs) excel at complex density estimation, their inherent nonlinearity creates a mismatch with the linear Bellman equation. Bellman Diffusion addresses this by modeling the gradient and scalar fields of a distribution directly, preserving linearity and enabling efficient integration with RL frameworks.
The Nonlinearity Bottleneck
Modern DGMs like Energy-Based Models (EBMs) or score-matching diffusion models typically operate on log-densities or normalized scores. However, the Bellman equation—the cornerstone of temporal difference learning—is a linear operator. Applying nonlinear DGMs to recursive RL updates often requires expensive trajectory sampling or discrete approximations (e.g., C51). Bellman Diffusion overcomes this "nonlinearity bottleneck" by shifting the modeling target to fields that respect linear operations.
Methodology: Field Modeling and Sampling
Bellman Diffusion models the probability density $p(x)$ through its constituent fields: the scalar field $s_\varphi(x) \approx p(x)$ and the gradient field $g_\phi(x) \approx \nabla p(x)$. Since both the identity and gradient operators are linear, the Bellman update can be applied directly to these neural proxies.
To sample from the learned distribution, the authors propose a novel Stochastic Differential Equation (SDE): $$dx(t) = \nabla p_{target}(x(t)) dt + \sqrt{p_{target}(x(t))} dw(t)$$ Unlike standard Langevin dynamics, which uses the score function $\nabla \log p(x) = \frac{\nabla p(x)}{p(x)}$, this SDE avoids division by the density. This prevents numerical instability in low-density regions and allows the model to accurately capture unbalanced modes. To scale to high dimensions, the authors employ a "slice trick," projecting high-dimensional densities onto 1D random directions for efficient training.
Experimental Results
The framework was evaluated on synthetic benchmarks, UCI tabular datasets, and RL environments such as Frozen Lake and Cart Pole.
A key finding is the model's ability to handle discontinuous distributions. As shown in the figure above, Bellman Diffusion accurately learns uniform distributions over disjoint spans by capturing the sharp derivatives at boundaries—a task typically difficult for continuous neural networks. In RL tasks, Bellman Diffusion achieved 1.5x faster convergence than the widely-used C51 histogram baseline in the Cart Pole environment, demonstrating superior stability and efficiency.
Conclusion and Future Work
Bellman Diffusion provides a theoretically grounded bridge between generative modeling and RL. By prioritizing linearity, it enables high-fidelity return distribution estimation that integrates seamlessly with the Bellman equation. While currently sensitive to certain hyperparameters like the variance term ($\epsilon$), future work aims to scale the model to high-resolution image domains and develop discrete-time convergence theorems to further bridge the gap between theory and practice.
Summary updated