TL;DR
MeanFlow2Better (MF2B) reformulates offline reinforcement learning as a "quality uplift" problem, learning to map suboptimal trajectory chunks toward high-value ones in a compressed latent space. By utilizing a novel one-step mean flow matching objective, it achieves state-of-the-art performance on D4RL benchmarks while eliminating the high inference costs typically associated with multi-step diffusion and flow models.
What Problem Does This Paper Address?
Offline Reinforcement Learning (RL) is the art of learning a policy from a fixed dataset without any new environment interaction. This is crucial for real-world applications like robotics or healthcare, where trial-and-error is dangerous or expensive. However, two major hurdles remain:
- The Suboptimal Data Trap: Most real-world datasets are full of "mediocre" behavior. Standard methods often treat all data as equal or struggle to effectively "lift" these suboptimal trajectories into expert-level ones.
- The Inference Bottleneck: Modern generative policies (like those based on Diffusion) are highly expressive but slow. They require multiple iterative steps to generate an action, making them impractical for high-frequency control.
Methods
MF2B introduces a paradigm shift in how we model generative policies for RL:
- Chunk-Level Latent Decisions: Instead of modeling raw actions at every timestep, the authors use an encoder to compress trajectory chunks (length $H$) into a high-level latent decision space. This reduces the sequence length from $T$ to $K \approx T/H$, simplifying the learning problem.
- One-Step Mean Flow: Standard flow matching requires solving an Ordinary Differential Equation (ODE) over many steps. MF2B adopts Mean Flow, which learns to predict the average velocity between a starting point and a target. This allows the model to generate a refined decision in a single Euler step ($x_1 = x_0 + u(x_0)$).
- Dual-Branch Flow Mapping: The model learns two paths simultaneously:
- Source-to-Positive: Maps Gaussian noise (a "value-neutral" starting point) to general positive behaviors.
- Quality-Uplift: Explicitly learns the vector field that transports suboptimal trajectory chunks toward high-value ones.
- Hierarchical Advantage Guidance: The authors use an Implicit Q-Learning (IQL) critic to calculate "chunk-level advantage." This advantage acts as a weight, steering the flow learning toward regions of the state space that yield higher returns.
Datasets
The model was evaluated on the standard D4RL Gym-MuJoCo benchmark. This includes three environments (HalfCheetah, Hopper, and Walker2d) across three dataset types:
- Medium: Data from a partially trained policy.
- Medium-Replay: The entire replay buffer of a policy trained to medium performance.
- Medium-Expert: A mix of suboptimal and expert trajectories.
Key Insights and Ideas
- Suboptimal Data as a Starting Point: The authors argue that suboptimal trajectories aren't just "noise" to be filtered out; they are structured starting points for improvement. By pairing low-value and high-value trajectories, the model learns the specific "direction" of improvement.
- Latent Space Smoothness: Moving the flow matching to a compressed latent space makes the mapping more linear. This is what fundamentally enables high-fidelity one-step generation, which is usually too difficult to achieve in raw, high-frequency action space.
- Neural Null-Space Inspiration: Drawing from neuroscience, the authors use "Gaussian-Induced Decision Sources" as a semantic null-space. This provides a stable baseline for the model to begin sculpting task-relevant representations.
Key Results
MF2B sets a new bar for one-step generative policies in offline RL.
As shown in Table 1, MF2B achieved an average score of 90.91 across the 9 D4RL tasks. Key highlights include:
- Efficiency: It matches or surpasses the performance of multi-step diffusion models (like DQL) while being significantly faster at inference time.
- Superiority in Flow: It outperforms other one-step flow-based baselines like FlowQ (85.6) and SSCP (87.9).
- Robustness: It performed particularly well on "medium-replay" datasets, which are notoriously difficult because they contain a high volume of noisy, suboptimal data.
Limitations and Future Work
While MF2B is powerful, the authors note a few areas for growth:
- Dataset Dependency: The quality of the learned latent space is still bounded by the diversity of the offline dataset. If the dataset is too narrow, the model may struggle to generalize.
- Complex Priors: Future work could involve using Vision-Language Models (VLMs) to provide more grounded latent priors instead of simple Gaussian noise.
- Scaling: The authors suggest extending this framework to goal-conditioned RL and multi-task settings, where the ability to "uplift" behavior toward specific goals could be even more impactful.
Summary updated