Abstract
Off-policy reinforcement learning (RL) typically relies on learning a state-action value function (Q-function), which suffers from the curse of dimensionality in high-dimensional action spaces. We present Vlearn, an off-policy RL algorithm that utilizes a state-value function (V-function) instead. By applying weighted importance sampling to the entire Bellman error, Vlearn achieves stable and efficient learning. Our results on high-dimensional benchmarks, including the 38-DOF DeepMind Control Dog tasks and the 39-DOF MyoHand, show that Vlearn consistently outperforms state-of-the-art off-policy methods like SAC and MPO.
Introduction
Most modern off-policy RL algorithms, such as Soft Actor-Critic (SAC) or TD3, rely on learning a Q-function, $Q(s, a)$, which estimates the value of taking a specific action in a specific state. While powerful, this approach struggles in high-dimensional action spaces because the critic must learn over the joint space of states and actions. The amount of data required to obtain a reliable estimate grows exponentially with dimensionality—a classic case of the "curse of dimensionality."
On-policy methods often use simpler V-functions, $V(s)$, which only depend on the state, but they are generally data-inefficient as they cannot easily reuse old data. Existing attempts to bridge this gap, such as V-trace, were designed for "nearly" on-policy data and suffer from high variance or optimization biases in truly off-policy settings. Vlearn addresses these issues by adapting the state-value function for deep off-policy RL.
Methods
Vlearn introduces several key innovations to stabilize V-function estimation:
- Weighted Importance Sampling (WIS) Loss: Instead of reweighting only the Bellman targets, Vlearn applies importance weights to the entire squared Bellman error. This ensures optimization stability even when importance ratios are small.
- Trust Region Policy Updates: Vlearn uses Trust Region Projection Layers (TRPL) to ensure the policy does not change too drastically between updates, which is crucial for learning from off-policy data.
- Twin Value Networks: Vlearn employs two V-function networks to mitigate optimization bias and provide ensemble regularization.
- Advantage Estimation: Since there is no Q-function, the advantage is calculated using the one-step temporal difference: $A(s, a) = r + \gamma V(s') - V(s)$.
The intuition behind Vlearn's stability compared to V-trace is illustrated in Figure 1. V-trace shifts the optimum toward the target critic as the importance ratio drops, while Vlearn simply scales the loss, maintaining the correct optimum.
Experimental Evaluation
We evaluated Vlearn on a suite of challenging continuous control benchmarks with high-dimensional action spaces, including Gymnasium's Humanoid-v4 (17D) and DeepMind Control's "Dog" locomotion tasks (38D).
Vlearn consistently outperformed state-of-the-art baselines like SAC, MPO, and PPO. In the Humanoid-v4 task, Vlearn achieved a 25% increase in final reward over SAC. In the 38D Dog tasks, where SAC and MPO struggled to learn consistent policies, Vlearn learned stable, high-performing locomotion for walking, trotting, and running.
By removing the action from the critic's input, the network only needs to learn the "topography" of the state space. This makes the critic much more robust to the specific actions taken during exploration, leading to faster and more stable convergence in complex environments.
Conclusion
Vlearn demonstrates that state-value functions can be highly effective for off-policy RL in high-dimensional spaces. By leveraging a theoretically sound weighted importance sampling objective and trust-region updates, Vlearn achieves superior stability and performance compared to traditional Q-learning approaches. Future work will explore extending Vlearn to offline RL and overactuated systems.
Summary updated