TL;DR
Latent Policy Steering (LPS) eliminates the brittle hyperparameter tuning and lossy value-function distillation common in offline reinforcement learning. By backpropagating original action-space Q-gradients directly through a differentiable, one-step MeanFlow generative policy, LPS achieves robust, tuning-free policy improvement. This approach delivers state-of-the-art performance across simulation benchmarks and real-world robotic manipulation tasks.
What Problem Does This Paper Address?
Offline reinforcement learning (RL) allows robots to learn complex skills from pre-collected datasets without risky real-world exploration. However, a fundamental challenge is distributional shift: when maximizing rewards, the policy often selects actions outside the dataset's support, leading to catastrophic extrapolation errors from the critic.
Standard offline RL algorithms mitigate this by using explicit behavior regularization (e.g., TD3+BC, QC-FQL), balancing reward maximization and data adherence via a hyperparameter $\alpha$. As shown in Figure 2, this trade-off is incredibly brittle: a large $\alpha$ yields overly conservative behavior (cloning the dataset), while a small $\alpha$ fails to constrain the policy, leading to out-of-distribution actions.
Prior latent steering methods (like DSRL) attempt to solve this by optimizing actions within a learned, safe latent manifold. However, in a fully offline setting, they must distill the action-space critic $Q(s,a)$ into a proxy latent-space critic $Q(s,z)$. This distillation step is highly lossy and introduces a severe gradient mismatch near the boundaries of the data manifold (Figure 3), severely limiting offline performance.
Methods
The authors introduce Latent Policy Steering (LPS), a framework that structurally decouples behavioral constraints from reward maximization without relying on explicit regularization weights or proxy latent critics.
LPS achieves this through three core components:
- Differentiable Base Policy via MeanFlow: Instead of treating the generative base policy as a black box, LPS leverages MeanFlow—a one-step generative model that provides a mathematically differentiable mapping from latents to actions. A noise-to-action reformulation is used during training to directly predict denoised action chunks, ensuring numerical stability.
- Spherical Latent Geometry: To prevent "norm explosion"—a common failure mode where the latent actor outputs massive latent vectors to exploit unconstrained spaces—LPS constrains both the base policy prior and the latent actor's output to a synchronized hypersphere ($S^{d-1}$ with radius $\sqrt{d}$). This keeps queries strictly within the valid coverage of the base policy.
- Direct Latent Policy Steering: Because the base policy is differentiable, the latent actor can be optimized directly using the gradients of the original action-space critic $Q_\theta(s,a)$ via the chain rule. This bypasses the need for an inaccurate proxy latent critic entirely.
Datasets
The authors evaluated LPS on both simulated and real-world datasets:
- OGBench: A standard offline goal-conditioned RL benchmark. The authors utilized five state-based manipulation tasks (cube-single, cube-double, scene-sparse, puzzle-3x3-sparse, puzzle-4x4) and their pixel-based counterparts (visual-task).
- Real-World Robotic Dataset: The authors collected 50 human-teleoperated demonstrations per task using a Franka Research 3 robot (via the DROID platform setup) across four precision-critical tasks: pick and place carrots, eggplant to bin, plug in bulb, and refill the tape.
Key Insights and Ideas
- Decoupling over Regularization: Rather than tuning a fragile hyperparameter to balance rewards and constraints, freezing a generative model to define the safe manifold and letting the actor steer within it provides a much more robust, "out-of-the-box" solution.
- Value Matching $\neq$ Gradient Matching: Even if a distilled latent critic accurately approximates the values of an action-space critic, its gradients can point in completely wrong directions near data boundaries. Direct backpropagation through a one-step generative model preserves high-fidelity gradient flow.
- The Importance of Latent Geometry: Unconstrained latent spaces invite optimization instability. Forcing the latent space onto a hypersphere aligns perfectly with the concentration of measure property of high-dimensional Gaussians, ensuring the actor stays within the generative model's "typical set."
Key Results
- State-of-the-Art Simulation Performance: On OGBench, LPS consistently outperformed one-step distillation baselines (QC-FQL, QC-MFQL) and latent steering with distilled critics (DSRL). On the challenging cube-double task, DSRL failed completely due to gradient mismatch, while LPS maintained high success rates.
- Extreme Hyperparameter Robustness: In sensitivity sweeps where $\alpha$ was varied from 0.01 to 300, action-space regularized methods crashed rapidly outside a narrow optimal peak. LPS remained entirely stable across the entire range.
- Real-World Robotic Success: LPS achieved the highest average success rate (56.2%) across all real-world tasks, substantially outperforming behavioral cloning baselines (Flow-BC: 31.2%, MF-BC: 28.7%) and DSRL (35.0%). Notably, on the precision-critical plug in bulb task, DSRL achieved 0% success, whereas LPS achieved 35%.
- Computational Efficiency: LPS trains significantly faster than DSRL by avoiding iterative sampling and proxy distillation overhead. At inference time, it achieves a rapid latency of 5.18 ms, making it highly practical for real-time control.
Limitations and Future Work
Ultimately, LPS is bounded by the quality and coverage of its underlying base policy; if the generative model fails to capture a specific behavior mode in the dataset, latent steering cannot recover it. Additionally, the spherical constraint is intentionally conservative, which may restrict the policy from extrapolating to highly novel, high-reward behaviors far beyond the demonstration distribution.
For future work, the authors propose scaling LPS to large Vision-Language-Action (VLA) models for general-purpose manipulation and exploring structured latent representations to better exploit the temporal structure within action chunks.
Summary updated