Efficient Off-Policy Learning for High-Dimensional Action Spaces

Fabian Otto, Philipp Becker, Vien Ang Ngo, Gerhard Neumann

Abstract

Existing off-policy reinforcement learning algorithms often rely on an explicit state-action-value function representation, which can be problematic in high-dimensional action spaces due to the curse of dimensionality. This reliance results in data inefficiency as maintaining a state-action-value function in such spaces is challenging. We present an efficient approach that utilizes only a state-value function as the critic for off-policy deep reinforcement learning. This approach, which we refer to as Vlearn, effectively circumvents the limitations of existing methods by eliminating the necessity for an explicit state-action-value function. To this end, we leverage a weighted importance sampling loss for learning deep value functions from off-policy data. While this is common for linear methods, it has not been combined with deep value function networks. This transfer to deep methods is not straightforward and requires novel design choices such as robust policy updates, twin value function networks to avoid an optimization bias, and importance weight clipping. Our approach improves sample complexity as well as final performance and ensures consistent and robust performance across various benchmark tasks. Eliminating the state-action-value function in Vlearn facilitates a streamlined learning process, yielding high-return agents.

Introduction: The Challenge of High-Dimensional Action Spaces

In reinforcement learning (RL), off-policy algorithms like Soft Actor-Critic (SAC) or Maximum a Posteriori Policy Optimization (MPO) typically utilize a critic that learns a state-action value function, $Q(s, a)$. This function estimates the expected return of taking a specific action $a$ in state $s$. While effective for many tasks, this paradigm encounters significant hurdles in high-dimensional action spaces—a phenomenon often referred to as the "curse of dimensionality."

As the number of degrees of freedom increases (e.g., in complex robotics or musculoskeletal models), the action space grows exponentially. Learning an accurate $Q$-function becomes increasingly difficult because the agent can only sample a negligible fraction of the possible actions. This leads to high variance, data inefficiency, and training instability. Conversely, on-policy methods like Proximal Policy Optimization (PPO) avoid $Q$-functions by using a state-value function $V(s)$, but they are generally less sample-efficient as they cannot effectively reuse historical data from a replay buffer.

The Vlearn Algorithm

To bridge this gap, we introduce Vlearn, an off-policy RL algorithm that relies solely on a state-value function $V(s)$ as its critic. By focusing on $V(s)$, the critic learns a more general representation of state quality, independent of the specific high-dimensional action taken.

Vlearn adapts Weighted Importance Sampling (WIS) for deep neural networks. Key technical innovations include:

  • WIS Bellman Loss: Unlike V-trace, which reweights only the target value, Vlearn applies importance weights to the entire Bellman error. This approach provides an upper bound on the true objective and maintains a consistent optimum.
  • Trust Region Projection Layers (TRPL): To ensure stable policy updates, Vlearn employs TRPL, which mathematically enforces a trust region more robustly than standard clipping methods.
  • Twin Value Networks: Vlearn utilizes two $V$-networks to mitigate optimization bias, a technique borrowed from $Q$-learning.
  • Importance Weight Clipping: Truncating importance ratios prevents destabilizing gradients from highly off-policy data.

Theoretical Insights: Addressing the "Optimum Shift"

A critical advantage of Vlearn over existing methods like V-trace is its robustness to off-policy data. In V-trace, when data is highly off-policy (low importance ratio $\rho$), the value estimate shifts toward the current target critic rather than the true Bellman target. Vlearn avoids this "optimum shift" by simply scaling the loss, ensuring the objective remains centered on the correct target.

Comparison of Vlearn and V-trace losses under varying importance ratios $\rho$.

Furthermore, analysis shows that the WIS estimator used in Vlearn typically exhibits lower variance compared to the squared estimators used in V-trace, contributing to more stable training.

Experimental Evaluation

We evaluated Vlearn against several state-of-the-art baselines across a range of continuous control benchmarks, including Gymnasium (Ant, Humanoid), DeepMind Control (DMC) Dog, and MyoSuite musculoskeletal models.

Performance comparison of Vlearn against baselines across Gymnasium and DeepMind Control Dog tasks.

Vlearn demonstrates its greatest strengths as task complexity increases:

  • High-Dimensional Dominance: On the 38-dimensional DMC Dog tasks, standard off-policy methods like SAC and MPO often failed to learn consistent policies. Vlearn, however, learned reliably across walk, trot, and run behaviors.
  • Humanoid Performance: Vlearn achieved a 25% increase in return over SAC on the 17-dimensional Humanoid-v4 task.
  • Musculoskeletal Control: In the MyoHand environment (39-dimensional), Vlearn achieved the highest aggregated success rate across ten distinct tasks, highlighting its efficacy in biologically inspired control.

Conclusion

Vlearn demonstrates that an explicit state-action-value function is not necessary for high-performance off-policy RL. By leveraging a robustly learned state-value function, Vlearn circumvents the curse of dimensionality in high-dimensional action spaces, offering a more stable and sample-efficient alternative for complex robotic and biological control tasks.

Summary updated

Abstract

Existing off-policy reinforcement learning algorithms often rely on an explicit state-action-value function representation, which can be problematic in high-dimensional action spaces due to the curse of dimensionality. This reliance results in data inefficiency as maintaining a state-action-value function in such spaces is challenging. We present an efficient approach that utilizes only a state-value function as the critic for off-policy deep reinforcement learning. This approach, which we refer to as Vlearn, effectively circumvents the limitations of existing methods by eliminating the necessity for an explicit state-action-value function. To this end, we leverage a weighted importance sampling loss for learning deep value functions from off-policy data. While this is common for linear methods, it has not been combined with deep value function networks. This transfer to deep methods is not straightforward and requires novel design choices such as robust policy updates, twin value function networks to avoid an optimization bias, and importance weight clipping. We also present a novel analysis of the variance of our estimate compared to commonly used importance sampling estimators such as V-trace. Our approach improves sample complexity as well as final performance and ensures consistent and robust performance across various benchmark tasks. Eliminating the state-action-value function in Vlearn facilitates a streamlined learning process, yielding high-return agents.
BibTeX
@inproceedings{otto2025efficient,
  author = {Fabian Otto and Philipp Becker and Vien Ang Ngo and Gerhard Neumann},
  title = {{Efficient Off-Policy Learning for High-Dimensional Action Spaces}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=JDzTI9rKls},
}
Figure