Latent Policy Steering through One-Step Flow Policies

Hokyun Im, Chenfeng Xu, Jianlong Fu, Yea Seol Lee

Abstract

Offline reinforcement learning (RL) allows robots to learn from offline datasets without risky exploration. Yet, offline RL's performance often hinges on a brittle trade-off between (1) return maximization, which can push policies outside the dataset support, and (2) behavioral constraints, which typically require sensitive hyperparameter tuning. Latent steering offers a structural way to stay within the dataset support during RL, but existing offline adaptations commonly approximate action values using latent-space critics learned via indirect distillation, which can lose information and hinder convergence. We propose Latent Policy Steering (LPS), which enables high-fidelity latent policy improvement by backpropagating original-action-space Q-gradients through a differentiable one-step MeanFlow policy to update a latent-action-space actor. By eliminating proxy latent critics, LPS allows an original-action-space critic to guide end-to-end latent-space optimization, while the one-step MeanFlow policy serves as a behavior-constrained generative prior. This decoupling yields a robust method that works out-of-the-box with minimal tuning. Across OGBench and real-world robotic tasks, LPS achieves state-of-the-art performance and consistently outperforms behavioral cloning and strong latent steering baselines.

Introduction

Offline Reinforcement Learning (RL) is often considered the "holy grail" of robotics, promising the ability to teach robots complex tasks using only pre-recorded datasets. This avoids the high risks and costs associated with real-world trial and error. However, practitioners face a fundamental "brittle trade-off": a robot must maximize rewards (return maximization) while strictly adhering to the behaviors observed in the dataset (behavioral constraints).

Traditional methods balance these objectives using a weighting hyperparameter ($\alpha$). If $\alpha$ is too small, the robot may attempt "hallucinated" actions outside the data support; if it is too large, the robot merely mimics the data (Behavioral Cloning) without improvement. Finding the optimal $\alpha$ requires extensive tuning that is often impossible in real-world deployments. Latent Policy Steering (LPS) addresses this by replacing delicate hyperparameter balancing with a structural constraint.

The LPS Framework

LPS decouples the "safe manifold" of behaviors from the "steering" mechanism. It utilizes a differentiable one-step MeanFlow policy as a behavior-constrained generative prior. Unlike complex, iterative diffusion models, MeanFlow produces actions in a single step, allowing the system to "look through" the generative model to see how changing a latent variable affects the final action and its predicted value.

Comparison of LPS architecture against common offline RL baselines QC-FQL and DSRL.

Key technical components include:

  • Direct Gradient Backpropagation: Unlike previous methods that "distill" a proxy value function into the latent space, LPS backpropagates gradients from the high-fidelity action-space critic directly to the latent actor. This ensures the improvement signal remains as accurate as possible.
  • Spherical Latent Geometry: To prevent "norm explosion" where optimizers push latent variables to extreme values, LPS constrains the latent space to a hypersphere. This keeps the robot's queries within the "typical set" of behaviors it has actually learned.
  • Action Chunking: LPS predicts sequences of actions (chunks) rather than single steps, providing the temporal coherence necessary for smooth robotic movement.

High-Fidelity Latent Optimization

A core insight of LPS is that eliminating proxy latent critics significantly improves the quality of the policy improvement signal. Previous latent methods often struggled because their distilled latent critics were essentially "blurry" approximations of the true value function. By using the original action-space critic, LPS ensures that the steering signal is high-fidelity.

Visualization of Q-value landscapes and gradient alignment, demonstrating how LPS avoids the lossy distillation issues of previous latent methods.

Experimental Evaluation

The researchers evaluated LPS across a comprehensive suite of simulated and real-world tasks:

  • OGBench: On this benchmark of simulated manipulation tasks (e.g., cube-double, puzzle-4x4), LPS consistently outperformed strong baselines like QC-FQL and DSRL.
  • Real-World Robotics: In physical trials on a Franka Research 3 robot—covering tasks like pick and place carrots and plug in bulb—LPS achieved a 56.2% average success rate, significantly outperforming Behavioral Cloning (~30%) and DSRL (35.0%).
  • Efficiency and Robustness: LPS is computationally efficient, with training speeds faster than DSRL and inference latencies (~5ms) comparable to simple BC models. Crucially, LPS remained stable across a wide range of settings where other methods saw performance crashes due to hyperparameter sensitivity.

Conclusion

Latent Policy Steering provides a robust, "tuning-free" approach to offline RL by structurally enforcing behavioral constraints through a differentiable generative prior. While the method is ultimately bounded by the quality of the base policy's training data, it represents a significant step toward reliable, high-performance robotic learning. Future work will explore scaling LPS to large Vision-Language-Action (VLA) models and more structured latent representations.

Summary updated

Abstract

Offline reinforcement learning (RL) allows robots to learn from offline datasets without risky exploration. Yet, offline RL's performance often hinges on a brittle trade-off between (1) return maximization, which can push policies outside the dataset support, and (2) behavioral constraints, which typically require sensitive hyperparameter tuning. Latent steering offers a structural way to stay within the dataset support during RL, but existing offline adaptations commonly approximate action values using latent-space critics learned via indirect distillation, which can lose information and hinder convergence. We propose Latent Policy Steering (LPS), which enables high-fidelity latent policy improvement by backpropagating original-action-space Q-gradients through a differentiable one-step MeanFlow policy to update a latent-action-space actor. By eliminating proxy latent critics, LPS allows an original-action-space critic to guide end-to-end latent-space optimization, while the one-step MeanFlow policy serves as a behavior-constrained generative prior. This decoupling yields a robust method that works out-of-the-box with minimal tuning. Across OGBench and real-world robotic tasks, LPS achieves state-of-the-art performance and consistently outperforms behavioral cloning and strong latent steering baselines.
BibTeX
@misc{im2026latent,
  author = {Hokyun Im and Chenfeng Xu and Jianlong Fu and Yea Seol Lee},
  title = {{Latent Policy Steering through One-Step Flow Policies}},
  year = {2026},
  eprint = {2603.05296},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.05296},
}
Figure