Conditional Flow Models for Offline Maximum Entropy Reinforcement Learning

Hao Cheng, Yuling Jiao, Lican Kang, Jerry Zhijian Yang, Chen ZhongarXiv 2026 · arXiv

TL;DR

This paper introduces a "separate and conquer" framework for offline reinforcement learning that uses conditional probability flow models to explicitly learn environment dynamics and rewards. By decoupling environment modeling from value function learning, the authors achieve state-of-the-art performance on offline benchmarks and provide strong theoretical guarantees without requiring the restrictive "Bellman completeness" assumption.

What Problem Does This Paper Address?

In Offline Reinforcement Learning (RL), an agent must learn an optimal strategy from a fixed dataset without the luxury of interacting with the environment to "try things out." Maximum Entropy (MaxEnt) RL adds a layer of complexity by encouraging stochastic policies that maximize both reward and entropy, leading to more robust behavior.

The core challenge lies in the Bellman equation. To estimate the value of an action ($Q^*$), you need to calculate an expectation over the next state and the reward. However, in an offline setting, the transition kernel (how the world moves) and the reward distribution are unknown. Most existing methods either try to learn these jointly with the value function—which can lead to unstable, coupled optimization—or use simplified models (like Gaussians) that fail to capture complex, multimodal real-world dynamics.

Methods

The authors propose a two-stage framework that treats environment modeling and value learning as distinct tasks:

  1. Conditional Flow Learning: Instead of just predicting the "average" next state, the researchers use Conditional Probability Flow ODEs (a type of score-based generative model) to estimate the full conditional distributions of rewards and transitions. This allows the model to represent complex, multimodal distributions (e.g., a robot that could turn either left or right to avoid an obstacle).
  2. Soft Bellman Residual Minimization: With the environment models "frozen," they construct a data-driven approximation of the soft Bellman operator. They then estimate the optimal soft action-value function ($Q^*$) by minimizing the empirical Bellman residual using a Deep Neural Network (DNN).

What’s new? While "Flow-based RL" has been used before to model policies, this paper is one of the first to use flows to model the environment side in a MaxEnt setting. This separation allows for a cleaner theoretical analysis and avoids the "moving target" problem common in joint optimization.

Datasets

The method was evaluated across three distinct settings:

  • Simulated Multimodal MDP: A custom-built environment specifically designed with multimodal transitions and rewards to test the flow model's expressive power.
  • Classical Control Tasks: Standard benchmarks including Pendulum, a stochastic version of the Inverted Pendulum, and a five-mode stochastic MountainCar.
  • D4RL HalfCheetah: A standard offline RL benchmark using four dataset types: Random, Medium, Medium-Replay, and Full-Replay.

Key Insights and Ideas

  • Smoothness Inheritance: A major theoretical contribution is showing that $Q^*$ "inherits" its smoothness (Hölder regularity) directly from the environment's reward and transition densities. This means if the world is smooth, the value function is smooth, which justifies using neural networks for estimation.
  • Bypassing Bellman Completeness: Most theoretical proofs in Deep RL rely on "Bellman completeness"—the assumption that your neural network class is perfectly closed under the Bellman operator. This paper bypasses this unrealistic assumption by estimating the operator before performing the residual minimization.
  • Localized Modeling: Unlike some model-based methods that generate long "imagined" trajectories (which accumulate error), this framework uses the learned model only for one-step expectations, making it more robust to model bias.

Key Results

The proposed method, simply called "Ours" in the paper, consistently outperformed strong baselines like SAC, CQL, and MOPO:

  • Simulated MDP: The method achieved a return of 861.69, nearly doubling the performance of SAC (533.48) and CQL (504.31).
  • D4RL HalfCheetah: It achieved an average normalized score of 52.32, beating the strongest baseline (CQL at 48.05).

  • Consistency: On the "Medium" dataset—often the most challenging because it contains useful but limited behavior—the method showed a massive gain, scoring 64.03 compared to CQL's 49.42 and SAC's 20.04.

Limitations and Future Work

  • Dimensionality: While successful on HalfCheetah, the authors note that future work should examine how this approach scales to even higher-dimensional continuous control tasks (e.g., full humanoid robots).
  • Partial Observability: The current framework assumes a fully observable Markov Decision Process (MDP). Extending this to partially observed environments (POMDPs) would be a significant next step.
  • Policy Optimization: The paper focuses on estimating the value function ($Q^*$). Integrating this into a full offline policy optimization pipeline is a promising direction for future research.

Summary updated

Abstract

Offline maximum entropy reinforcement learning seeks to estimate the optimal soft action-value function \(Q^*\) from a fixed dataset without further interaction with the environment. Although \(Q^*\) is characterized by the soft optimal Bellman equation, the corresponding Bellman operator cannot be evaluated directly because both the reward distribution and the transition kernel are unknown. Consequently, estimating \(Q^*\) requires the joint treatment of environment modeling and value function learning. We propose a framework that addresses these tasks separately. Conditional diffusion models based on probability flow ODEs estimate the conditional reward distribution and transition kernel. The resulting distributional estimators define a data driven approximation of the soft optimal Bellman operator, whose empirical Bellman residual is minimized over a class of deep neural networks to estimate \(Q^*\). We establish nonasymptotic guarantees for the proposed method. For operator learning, we derive total variation error bounds for the conditional flow estimators that account explicitly for score estimation, numerical discretization, and terminal smoothing. For soft value learning, we first use the soft Bellman equation to show that $Q^*$ inherits Hölder regularity from the conditional reward and transition densities. For any Hölder index $β$, the value learning stage achieves an excess Bellman residual risk bound of order $\widetilde{\mathcal O}\big(n^{-\frac{2β}{d+2β}}\big),$ where $d$ is the total dimension of the state and action spaces. In contrast to many existing theoretical analyses of deep reinforcement learning, our results do not require Bellman completeness. Numerical experiments support the theoretical findings and demonstrate the effectiveness of the proposed method.
BibTeX
@misc{cheng2026conditional,
  author = {Hao Cheng and Yuling Jiao and Lican Kang and Jerry Zhijian Yang and Chen Zhong},
  title = {{Conditional Flow Models for Offline Maximum Entropy Reinforcement Learning}},
  year = {2026},
  eprint = {2610.32125},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.32125},
}
Figure