Latent Policy Steering through One-Step Flow Policies

Hokyun Im, Andrey Kolobov, Jianlong Fu, Youngwoon Lee

TL;DR

Latent Policy Steering (LPS) eliminates the brittle hyperparameter tuning and lossy value-function distillation common in offline reinforcement learning. By backpropagating original action-space Q-gradients directly through a differentiable, one-step MeanFlow generative policy, LPS achieves robust, tuning-free policy improvement. This approach delivers state-of-the-art performance across simulation benchmarks and real-world robotic manipulation tasks.

What Problem Does This Paper Address?

Offline reinforcement learning (RL) allows robots to learn complex skills from pre-collected datasets without risky real-world exploration. However, a fundamental challenge is distributional shift: when maximizing rewards, the policy often selects actions outside the dataset's support, leading to catastrophic extrapolation errors from the critic.

Standard offline RL algorithms mitigate this by using explicit behavior regularization (e.g., TD3+BC, QC-FQL), balancing reward maximization and data adherence via a hyperparameter $\alpha$. As shown in Figure 2, this trade-off is incredibly brittle: a large $\alpha$ yields overly conservative behavior (cloning the dataset), while a small $\alpha$ fails to constrain the policy, leading to out-of-distribution actions.

Fig. 2: Sensitivity to the regularization weight α in FQL. Learned policy densities on a 2D toy task with reward concentrated in the top-right corner reveal a pattern: large α yields overly conservative policies, while small α encourages out-of-support actions.

Prior latent steering methods (like DSRL) attempt to solve this by optimizing actions within a learned, safe latent manifold. However, in a fully offline setting, they must distill the action-space critic $Q(s,a)$ into a proxy latent-space critic $Q(s,z)$. This distillation step is highly lossy and introduces a severe gradient mismatch near the boundaries of the data manifold (Figure 3), severely limiting offline performance.

Fig. 3: Comparing action space Q-value and distilled latent- space Q-value. Left to right: (1) dataset distribution with reward intensity; (2) action-space Q-value Qϕ(s, a) projected into the latent space; (3) learned latent Q-value Qϕ(s, z); (4) cosine similarity between the gradients in (2) and (3).

Methods

The authors introduce Latent Policy Steering (LPS), a framework that structurally decouples behavioral constraints from reward maximization without relying on explicit regularization weights or proxy latent critics.

Fig. 1: Comparison of policy extraction paradigms. (Top) QC-FQL constrains the policy via an explicit regularizer, cre- ating a trade-off between reward maximization and behavioral regularization. (Middle) DSRL resolves this trade-off via latent steering, but requires learning a latent-space critic Q(s, z) via distillation in the offline RL setting. (Bottom) LPS (Ours) achieves robust, tuning-free optimization by backpropagating action-space critic gradients ∇aQ(s, a) through a differentiable one-step generative policy.

LPS achieves this through three core components:

  1. Differentiable Base Policy via MeanFlow: Instead of treating the generative base policy as a black box, LPS leverages MeanFlow—a one-step generative model that provides a mathematically differentiable mapping from latents to actions. A noise-to-action reformulation is used during training to directly predict denoised action chunks, ensuring numerical stability.
  2. Spherical Latent Geometry: To prevent "norm explosion"—a common failure mode where the latent actor outputs massive latent vectors to exploit unconstrained spaces—LPS constrains both the base policy prior and the latent actor's output to a synchronized hypersphere ($S^{d-1}$ with radius $\sqrt{d}$). This keeps queries strictly within the valid coverage of the base policy.
  3. Direct Latent Policy Steering: Because the base policy is differentiable, the latent actor can be optimized directly using the gradients of the original action-space critic $Q_\theta(s,a)$ via the chain rule. This bypasses the need for an inaccurate proxy latent critic entirely.

Datasets

The authors evaluated LPS on both simulated and real-world datasets:

  • OGBench: A standard offline goal-conditioned RL benchmark. The authors utilized five state-based manipulation tasks (cube-single, cube-double, scene-sparse, puzzle-3x3-sparse, puzzle-4x4) and their pixel-based counterparts (visual-task).
  • Real-World Robotic Dataset: The authors collected 50 human-teleoperated demonstrations per task using a Franka Research 3 robot (via the DROID platform setup) across four precision-critical tasks: pick and place carrots, eggplant to bin, plug in bulb, and refill the tape.

Key Insights and Ideas

  • Decoupling over Regularization: Rather than tuning a fragile hyperparameter to balance rewards and constraints, freezing a generative model to define the safe manifold and letting the actor steer within it provides a much more robust, "out-of-the-box" solution.
  • Value Matching $\neq$ Gradient Matching: Even if a distilled latent critic accurately approximates the values of an action-space critic, its gradients can point in completely wrong directions near data boundaries. Direct backpropagation through a one-step generative model preserves high-fidelity gradient flow.
  • The Importance of Latent Geometry: Unconstrained latent spaces invite optimization instability. Forcing the latent space onto a hypersphere aligns perfectly with the concentration of measure property of high-dimensional Gaussians, ensuring the actor stays within the generative model's "typical set."

Key Results

  • State-of-the-Art Simulation Performance: On OGBench, LPS consistently outperformed one-step distillation baselines (QC-FQL, QC-MFQL) and latent steering with distilled critics (DSRL). On the challenging cube-double task, DSRL failed completely due to gradient mismatch, while LPS maintained high success rates.
  • Extreme Hyperparameter Robustness: In sensitivity sweeps where $\alpha$ was varied from 0.01 to 300, action-space regularized methods crashed rapidly outside a narrow optimal peak. LPS remained entirely stable across the entire range.
  • Real-World Robotic Success: LPS achieved the highest average success rate (56.2%) across all real-world tasks, substantially outperforming behavioral cloning baselines (Flow-BC: 31.2%, MF-BC: 28.7%) and DSRL (35.0%). Notably, on the precision-critical plug in bulb task, DSRL achieved 0% success, whereas LPS achieved 35%.
  • Computational Efficiency: LPS trains significantly faster than DSRL by avoiding iterative sampling and proxy distillation overhead. At inference time, it achieves a rapid latency of 5.18 ms, making it highly practical for real-time control.

Limitations and Future Work

Ultimately, LPS is bounded by the quality and coverage of its underlying base policy; if the generative model fails to capture a specific behavior mode in the dataset, latent steering cannot recover it. Additionally, the spherical constraint is intentionally conservative, which may restrict the policy from extrapolating to highly novel, high-reward behaviors far beyond the demonstration distribution.

For future work, the authors propose scaling LPS to large Vision-Language-Action (VLA) models for general-purpose manipulation and exploring structured latent representations to better exploit the temporal structure within action chunks.

Summary updated

Abstract

Offline reinforcement learning (RL) allows robots to learn from offline datasets without risky exploration. Yet, offline RL's performance often hinges on a brittle trade-off between (1) return maximization, which can push policies outside the dataset support, and (2) behavioral constraints, which typically require sensitive hyperparameter tuning. Latent steering offers a structural way to stay within the dataset support during RL, but existing offline adaptations commonly approximate action values using latent-space critics learned via indirect distillation, which can lose information and hinder convergence. We propose Latent Policy Steering (LPS), which enables high-fidelity latent policy improvement by backpropagating original-action-space Q-gradients through a differentiable one-step MeanFlow policy to update a latent-action-space actor. By eliminating proxy latent critics, LPS allows an original-action-space critic to guide end-to-end latent-space optimization, while the one-step MeanFlow policy serves as a behavior-constrained generative prior. This decoupling yields a robust method that works out-of-the-box with minimal tuning. Across OGBench and real-world robotic tasks, LPS achieves state-of-the-art performance and consistently outperforms behavioral cloning and strong latent steering baselines.
BibTeX
@misc{im2026latent,
  author = {Hokyun Im and Andrey Kolobov and Jianlong Fu and Youngwoon Lee},
  title = {{Latent Policy Steering through One-Step Flow Policies}},
  year = {2026},
  eprint = {2603.05296},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.05296},
}
Figure