Revisiting Gaussian mixture critics in off-policy reinforcement learning: a sample-based approach

Bobak Shahriari, Abbas Abdolmaleki, Arunkumar Byravan, Abe Friesen, Siqi Liu, Jost Tobias Springenberg, Nicolas Heess, Matt Hoffman, Martin A. RiedmillerarXiv 2022 · DOI · OpenReview

Abstract

Distributional Reinforcement Learning (RL) has demonstrated significant performance gains by predicting the full probability distribution of future returns rather than just their expected value. However, the widely used categorical critic (C51) requires manually-tuned reward boundaries ($v_{min}$ and $v_{max}$), which limits its flexibility in complex or unknown environments. This paper revisits the use of Mixture of Gaussians (MoG) as a representation for return distributions. By employing a sample-based cross-entropy loss, the proposed MoG critic eliminates the need for rigid support boundaries and achieves state-of-the-art performance on challenging continuous control tasks within the DeepMind Control Suite.

Introduction: The Challenge of Fixed Supports

In Distributional RL, the agent's goal is to model the distribution of returns. The most prevalent method, the categorical critic, represents this distribution as a histogram over a fixed set of "bins." While effective, this approach has a critical weakness: researchers must pre-specify the range of possible rewards. If these hyperparameters are misaligned with the actual reward scale of the environment, the agent's learning can be severely degraded. This is particularly problematic in real-world robotics or complex simulations where reward scales are not known a priori.

To address this, we seek a more flexible, "parameter-free" alternative that can adapt its support and resolution automatically as the agent learns. We propose a sample-based Mixture of Gaussians (MoG) critic that provides infinite support and adaptive resolution.

Methodology

The core of our approach is replacing the fixed histogram of a categorical critic with a Mixture of Gaussians. Instead of a discrete set of bins, the critic outputs the parameters—means, variances, and weights—of several Gaussian distributions.

Comparison of non-distributional, categorical (C51), and Mixture of Gaussians (MoG) critics.

Sample-based Loss

Unlike previous attempts at MoG critics that relied on complex mathematical distances, we utilize a simple Monte Carlo approximation to compute the cross-entropy between the current distribution and the target distribution generated by the distributional Bellman operator. This sample-based loss allows the model to remain computationally efficient while maintaining high representational capacity.

Integration with Modern Frameworks

The MoG critic is designed as a general-purpose upgrade for off-policy RL. We evaluated its performance when integrated with:

  • Maximum a Posteriori Policy Optimization (MPO): Creating a highly stable learner for complex tasks.
  • Deep Deterministic Policy Gradient (DDPG): Providing a more flexible version of the D4PG algorithm.

Experimental Evaluation

We evaluated the MoG critic on the DeepMind Control Suite, covering 25 different environments. The study specifically focused on "hard" tasks that typically challenge standard critics, such as the Humanoid (run, walk, stand), Dog (fetch), and Manipulator (insert ball) domains.

Learning curves comparing C51, MoG, and non-distributional agents across challenging domains.

Key Results

  • Superior Performance on Complex Tasks: As shown in the figures above, the MoG critic significantly outperforms the C51 baseline on the most difficult tasks, particularly in the Humanoid and Manipulator domains.
  • Hyperparameter Robustness: The model is remarkably robust to the number of Gaussian components. While at least one component is necessary for complex tasks, performance remains stable whether using 3, 5, or 10 components.
  • Algorithmic Agnostic Benefits: The performance gains were consistent across both MPO and DDPG frameworks, suggesting that the MoG critic is a robust, general-purpose improvement for off-policy RL.

Discussion and Key Insights

Our analysis suggests that the success of the MoG critic stems from two primary factors beyond simple distribution modeling:

  1. Enhanced Representation Learning: Requiring the neural network to predict multiple moments (mean and variance) serves as a powerful auxiliary task. This forces the shared network "torso" to learn richer, more descriptive features of the environment.
  2. Adaptive Reweighting: The loss function naturally performs a form of "Mahalanobis reweighting." It assigns higher weight to transitions where the model is certain (low variance) and less weight to noisy or uncertain transitions, leading to more stable updates.

Limitations and Future Work

While the MoG critic removes the need for $v_{min}$ and $v_{max}$, it introduces a sensitivity to the initial scale (variance) of the Gaussians. In manipulation tasks, initializing these scales to a very small value was crucial to prevent early divergence.

Furthermore, while the cross-entropy loss is effective in practice, it does not technically guarantee a mathematical "contraction." Future research could explore integrating the MoG framework with the Cramér distance or 2-Wasserstein distance to provide stronger theoretical convergence guarantees while maintaining the flexibility of the sample-based approach.

Summary updated

Abstract

Actor-critic algorithms that make use of distributional policy evaluation have frequently been shown to outperform their non-distributional counterparts on many challenging control tasks. Examples of this behavior include the D4PG and DMPO algorithms as compared to DDPG and MPO, respectively [Barth-Maron et al., 2018; Hoffman et al., 2020]. However, both agents rely on the C51 critic for value estimation.One major drawback of the C51 approach is its requirement of prior knowledge about the minimum andmaximum values a policy can attain as well as the number of bins used, which fixes the resolution ofthe distributional estimate. While the DeepMind control suite of tasks utilizes standardized rewards and episode lengths, thus enabling the entire suite to be solved with a single setting of these hyperparameters, this is often not the case. This paper revisits a natural alternative that removes this requirement, namelya mixture of Gaussians, and a simple sample-based loss function to train it in an off-policy regime. We empirically evaluate its performance on a broad range of continuous control tasks and demonstrate that it eliminates the need for these distributional hyperparameters and achieves state-of-the-art performance on a variety of challenging tasks (e.g. the humanoid, dog, quadruped, and manipulator domains). Finallywe provide an implementation in the Acme agent repository.
BibTeX
@misc{shahriari2022revisiting,
  author = {Bobak Shahriari and Abbas Abdolmaleki and Arunkumar Byravan and Abe Friesen and Siqi Liu and Jost Tobias Springenberg and Nicolas Heess and Matt Hoffman and Martin A. Riedmiller},
  title = {{Revisiting Gaussian mixture critics in off-policy reinforcement learning: a sample-based approach}},
  year = {2022},
  eprint = {2204.10256},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.10256},
}
Figure