Abstract
Distributional Reinforcement Learning (RL) has demonstrated significant performance gains by predicting the full probability distribution of future returns rather than just their expected value. However, the widely used categorical critic (C51) requires manually-tuned reward boundaries ($v_{min}$ and $v_{max}$), which limits its flexibility in complex or unknown environments. This paper revisits the use of Mixture of Gaussians (MoG) as a representation for return distributions. By employing a sample-based cross-entropy loss, the proposed MoG critic eliminates the need for rigid support boundaries and achieves state-of-the-art performance on challenging continuous control tasks within the DeepMind Control Suite.
Introduction: The Challenge of Fixed Supports
In Distributional RL, the agent's goal is to model the distribution of returns. The most prevalent method, the categorical critic, represents this distribution as a histogram over a fixed set of "bins." While effective, this approach has a critical weakness: researchers must pre-specify the range of possible rewards. If these hyperparameters are misaligned with the actual reward scale of the environment, the agent's learning can be severely degraded. This is particularly problematic in real-world robotics or complex simulations where reward scales are not known a priori.
To address this, we seek a more flexible, "parameter-free" alternative that can adapt its support and resolution automatically as the agent learns. We propose a sample-based Mixture of Gaussians (MoG) critic that provides infinite support and adaptive resolution.
Methodology
The core of our approach is replacing the fixed histogram of a categorical critic with a Mixture of Gaussians. Instead of a discrete set of bins, the critic outputs the parameters—means, variances, and weights—of several Gaussian distributions.
Sample-based Loss
Unlike previous attempts at MoG critics that relied on complex mathematical distances, we utilize a simple Monte Carlo approximation to compute the cross-entropy between the current distribution and the target distribution generated by the distributional Bellman operator. This sample-based loss allows the model to remain computationally efficient while maintaining high representational capacity.
Integration with Modern Frameworks
The MoG critic is designed as a general-purpose upgrade for off-policy RL. We evaluated its performance when integrated with:
- Maximum a Posteriori Policy Optimization (MPO): Creating a highly stable learner for complex tasks.
- Deep Deterministic Policy Gradient (DDPG): Providing a more flexible version of the D4PG algorithm.
Experimental Evaluation
We evaluated the MoG critic on the DeepMind Control Suite, covering 25 different environments. The study specifically focused on "hard" tasks that typically challenge standard critics, such as the Humanoid (run, walk, stand), Dog (fetch), and Manipulator (insert ball) domains.
Key Results
- Superior Performance on Complex Tasks: As shown in the figures above, the MoG critic significantly outperforms the C51 baseline on the most difficult tasks, particularly in the Humanoid and Manipulator domains.
- Hyperparameter Robustness: The model is remarkably robust to the number of Gaussian components. While at least one component is necessary for complex tasks, performance remains stable whether using 3, 5, or 10 components.
- Algorithmic Agnostic Benefits: The performance gains were consistent across both MPO and DDPG frameworks, suggesting that the MoG critic is a robust, general-purpose improvement for off-policy RL.
Discussion and Key Insights
Our analysis suggests that the success of the MoG critic stems from two primary factors beyond simple distribution modeling:
- Enhanced Representation Learning: Requiring the neural network to predict multiple moments (mean and variance) serves as a powerful auxiliary task. This forces the shared network "torso" to learn richer, more descriptive features of the environment.
- Adaptive Reweighting: The loss function naturally performs a form of "Mahalanobis reweighting." It assigns higher weight to transitions where the model is certain (low variance) and less weight to noisy or uncertain transitions, leading to more stable updates.
Limitations and Future Work
While the MoG critic removes the need for $v_{min}$ and $v_{max}$, it introduces a sensitivity to the initial scale (variance) of the Gaussians. In manipulation tasks, initializing these scales to a very small value was crucial to prevent early divergence.
Furthermore, while the cross-entropy loss is effective in practice, it does not technically guarantee a mathematical "contraction." Future research could explore integrating the MoG framework with the Cramér distance or 2-Wasserstein distance to provide stronger theoretical convergence guarantees while maintaining the flexibility of the sample-based approach.
Summary updated