From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learning

Shenghua Lu, Zixuan Yang, Rongxu Zhao, Bohao Qu, Jifeng Hu, Hechang ChenarXiv 2026 · arXiv

TL;DR

MeanFlow2Better (MF2B) reformulates offline reinforcement learning as a "quality uplift" problem, learning to map suboptimal trajectory chunks toward high-value ones in a compressed latent space. By utilizing a novel one-step mean flow matching objective, it achieves state-of-the-art performance on D4RL benchmarks while eliminating the high inference costs typically associated with multi-step diffusion and flow models.

What Problem Does This Paper Address?

Offline Reinforcement Learning (RL) is the art of learning a policy from a fixed dataset without any new environment interaction. This is crucial for real-world applications like robotics or healthcare, where trial-and-error is dangerous or expensive. However, two major hurdles remain:

  1. The Suboptimal Data Trap: Most real-world datasets are full of "mediocre" behavior. Standard methods often treat all data as equal or struggle to effectively "lift" these suboptimal trajectories into expert-level ones.
  2. The Inference Bottleneck: Modern generative policies (like those based on Diffusion) are highly expressive but slow. They require multiple iterative steps to generate an action, making them impractical for high-frequency control.

Methods

MF2B introduces a paradigm shift in how we model generative policies for RL:

  • Chunk-Level Latent Decisions: Instead of modeling raw actions at every timestep, the authors use an encoder to compress trajectory chunks (length $H$) into a high-level latent decision space. This reduces the sequence length from $T$ to $K \approx T/H$, simplifying the learning problem.
  • One-Step Mean Flow: Standard flow matching requires solving an Ordinary Differential Equation (ODE) over many steps. MF2B adopts Mean Flow, which learns to predict the average velocity between a starting point and a target. This allows the model to generate a refined decision in a single Euler step ($x_1 = x_0 + u(x_0)$).
  • Dual-Branch Flow Mapping: The model learns two paths simultaneously:
  • Source-to-Positive: Maps Gaussian noise (a "value-neutral" starting point) to general positive behaviors.
  • Quality-Uplift: Explicitly learns the vector field that transports suboptimal trajectory chunks toward high-value ones.
  • Hierarchical Advantage Guidance: The authors use an Implicit Q-Learning (IQL) critic to calculate "chunk-level advantage." This advantage acts as a weight, steering the flow learning toward regions of the state space that yield higher returns.

Datasets

The model was evaluated on the standard D4RL Gym-MuJoCo benchmark. This includes three environments (HalfCheetah, Hopper, and Walker2d) across three dataset types:

  • Medium: Data from a partially trained policy.
  • Medium-Replay: The entire replay buffer of a policy trained to medium performance.
  • Medium-Expert: A mix of suboptimal and expert trajectories.

Key Insights and Ideas

  • Suboptimal Data as a Starting Point: The authors argue that suboptimal trajectories aren't just "noise" to be filtered out; they are structured starting points for improvement. By pairing low-value and high-value trajectories, the model learns the specific "direction" of improvement.
  • Latent Space Smoothness: Moving the flow matching to a compressed latent space makes the mapping more linear. This is what fundamentally enables high-fidelity one-step generation, which is usually too difficult to achieve in raw, high-frequency action space.
  • Neural Null-Space Inspiration: Drawing from neuroscience, the authors use "Gaussian-Induced Decision Sources" as a semantic null-space. This provides a stable baseline for the model to begin sculpting task-relevant representations.

Key Results

MF2B sets a new bar for one-step generative policies in offline RL.

Table 1: D4RL Gym-MuJoCo Benchmark Comprehensive Comparison.

As shown in Table 1, MF2B achieved an average score of 90.91 across the 9 D4RL tasks. Key highlights include:

  • Efficiency: It matches or surpasses the performance of multi-step diffusion models (like DQL) while being significantly faster at inference time.
  • Superiority in Flow: It outperforms other one-step flow-based baselines like FlowQ (85.6) and SSCP (87.9).
  • Robustness: It performed particularly well on "medium-replay" datasets, which are notoriously difficult because they contain a high volume of noisy, suboptimal data.

Limitations and Future Work

While MF2B is powerful, the authors note a few areas for growth:

  • Dataset Dependency: The quality of the learned latent space is still bounded by the diversity of the offline dataset. If the dataset is too narrow, the model may struggle to generalize.
  • Complex Priors: Future work could involve using Vision-Language Models (VLMs) to provide more grounded latent priors instead of simple Gaussian noise.
  • Scaling: The authors suggest extending this framework to goal-conditioned RL and multi-task settings, where the ability to "uplift" behavior toward specific goals could be even more impactful.

Summary updated

Abstract

Offline reinforcement learning aims to learn effective policies from fixed datasets without environment interaction. Although generative models such as diffusion and flow matching enhance policy expressiveness, they typically start from uninformative Gaussian noise and generate actions through multi-step iteration, leading to high computational costs and failing to effectively leverage suboptimal trajectories in datasets. We propose MeanFlow2Better (MF2B), which reformulates offline RL as generative decision chunk optimization from current performance status to better performance status in a high-level latent decision space. MF2B maps trajectory chunks to high-level latent decision representations through an encoder, placing generation starting points within a structured decision space sculpted by behavioral data. The model learns a shared conditional mean velocity field in this space, constructing two flow mappings: from Gaussian-Induced Decision Sources toward positive behavioral decision regions, and from suboptimal decisions toward high-value decision regions. Both flow mappings complete decision refinement through one-step ODE integration over the unit time interval $Δt=1$, requiring neither multi-step iterative sampling nor backpropagation through time. Furthermore, we propose hierarchical chunk-level advantage to guide the flow direction toward high-return regions, where the high-level decisions cover multiple low-level actions, leading to sequence length compression from $T$ to $K \approx T/H$ and significantly lower computational overhead. On the D4RL Gym-MuJoCo benchmark, MF2B achieves an average score of 90.91, matching or surpassing strong baselines.
BibTeX
@misc{lu2026from,
  author = {Shenghua Lu and Zixuan Yang and Rongxu Zhao and Bohao Qu and Jifeng Hu and Hechang Chen},
  title = {{From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learning}},
  year = {2026},
  eprint = {2610.16984},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.16984},
}
Figure