Controllable Flow Matching for Online Reinforcement Learning

Bin Wang, Boxiang Tao, Haifeng Jing, Hongbo Dou, Zijian WangarXiv 2025 · DOI · arXiv

Introduction

Model-Based Reinforcement Learning (MBRL) offers a path toward high data efficiency by allowing agents to learn from a simulated "world model." However, traditional MBRL relies on autoregressive dynamics models that predict the next state $s_{t+1}$ based on the current state $s_t$ and action $a_t$. This sequential approach suffers from compounding errors: small inaccuracies in early predictions accumulate over time, leading to divergent "hallucinated" trajectories that degrade policy performance.

CtrlFlow addresses this fundamental bottleneck by introducing trajectory-level Conditional Flow Matching (CFM) to the online RL setting. Instead of step-by-step prediction, CtrlFlow generates entire sequences of states, actions, and rewards simultaneously, bypassing the Markovian chain of errors and achieving significant improvements in both learning speed and final performance.

Methodology

The CtrlFlow framework consists of three primary components designed to ensure that generated trajectories are accurate, physically plausible, and goal-oriented.

Trajectory-Level Generation

Unlike autoregressive models, CtrlFlow treats a full trajectory $\tau = \{(s, a, r)_1, \dots, (s, a, r)_H\}$ as a single sample from a high-dimensional distribution. By using CFM, the model learns to map simple noise to the complex distribution of valid environment trajectories. This holistic generation ensures distributional consistency across the entire time horizon $H$.

Controllability and Value Guidance

To ground the generative model, CtrlFlow incorporates two critical constraints:

  1. Controllability Gramian: Borrowing from control theory, the model utilizes a non-linear Controllability Gramian matrix to minimize the "control energy" required for transitions. This ensures that generated paths are dynamically sound and "steerable" within the environment's physical limits.
  2. Value-Guided Energy Fields: To bias generation toward high-reward outcomes, the vector fields are augmented with value guidance. This "pushes" the sampling process toward regions of the state space associated with higher cumulative returns.
The CtrlFlow architecture: (a) Training the flow model with controllability constraints, (b) Generating trajectories using value guidance and control energy, and (c) Improving the policy using sampled data.

Experimental Results

CtrlFlow was evaluated on standard MuJoCo continuous control benchmarks, including Hopper-v3, Walker2d-v3, HalfCheetah-v3, and InvertedPendulum-v4.

Efficiency and Performance

The results demonstrate that CtrlFlow significantly outpaces existing model-based and model-free methods. In the Hopper task, CtrlFlow reached 90% of its peak performance in approximately 35k steps—nearly twice as fast as ADMPO-ON (60k steps) and significantly faster than MPPVE (70k steps). Across all tasks, CtrlFlow provided a 2x to 4x speedup in convergence.

Comparative performance of CtrlFlow against baselines (ADMPO-ON, MPPVE, MACURA, MBPO, SAC) across four MuJoCo environments.

Beyond speed, CtrlFlow achieved higher asymptotic rewards. For instance, it stabilized at a reward of ~3300 in Hopper, whereas the model-free SAC baseline only reached ~1000. Ablation studies confirmed that the Controllability Gramian is vital for this stability; without it, rewards often failed to converge.

Discussion and Conclusion

CtrlFlow represents a paradigm shift in online MBRL by moving from sequential dynamics to trajectory-level generative modeling. By integrating control-theoretic principles with flow matching, it effectively mitigates compounding errors and enables zero-shot generalization of movement primitives across different tasks.

While highly effective in fully observable environments, future work is required to adapt CtrlFlow for Partially Observable Markov Decision Processes (POMDPs) and to extend the effective generation horizon beyond current limits (typically $H > 30$). Nonetheless, CtrlFlow sets a new benchmark for efficiency and stability in online reinforcement learning.

Summary updated

BibTeX
@misc{wang2025controllable,
  author = {Bin Wang and Boxiang Tao and Haifeng Jing and Hongbo Dou and Zijian Wang},
  title = {{Controllable Flow Matching for Online Reinforcement Learning}},
  year = {2025},
  eprint = {2511.06816},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.06816},
}
Figure