A Distributional Perspective on Actor-Critic Framework

Daniel Wontae Nam, Younghoon Kim, Chan Youn ParkOpenReview 2020 · OpenReview

Overview

GMAC (Gaussian Mixture Actor-Critic) is a novel framework that brings the benefits of distributional reinforcement learning to the popular actor-critic architecture. By modeling value distributions as Gaussian Mixture Models and using a new "Sample-Replacement" algorithm, it achieves superior performance across both discrete (Atari) and continuous (PyBullet) tasks with significantly lower computational overhead than previous distributional methods.

The Challenge with Distributional Actor-Critic Models

Standard Reinforcement Learning (RL) typically predicts a single scalar value: the expected (average) future return. Distributional RL (DRL) goes further by predicting the entire probability distribution of those returns. While DRL has shown massive success in value-based methods like DQN, it faces three major hurdles when moving to Actor-Critic frameworks:

  1. Instability: In control settings where the policy is constantly changing, distributional operators can fail to converge.
  2. Action Type Restrictions: Most DRL algorithms are designed specifically for either discrete or continuous action spaces, but rarely both.
  3. Conflation and Complexity: Common DRL methods (like Quantile Regression) often confuse samples with statistical summaries, requiring expensive "imputation" strategies to work correctly. They also tend to be computationally heavy.

GMAC is designed to solve these issues, providing a stable, efficient, and general-purpose distributional critic.

Core Innovations

The authors propose three key methodological improvements to make distributional actor-critic frameworks viable:

  • SR($\lambda$) (Sample-Replacement): To estimate the multi-step "$\lambda$-return" for a distribution, a naive approach would be computationally explosive ($O(N^2)$). SR($\lambda$) is a clever algorithm that generates these targets in a single sweep ($O(N)$) by replacing samples along a trajectory with a certain probability.
Diagram illustrating the Sample-Replacement algorithm over time steps
  • Cramér Distance & Energy Distance: Instead of the commonly used Wasserstein distance (which can lead to biased gradients), GMAC minimizes the Cramér distance. The authors utilize a mathematical link showing that the squared Cramér distance is equivalent to half the Energy Distance. This allows for unbiased gradients and more stable learning.
  • Gaussian Mixture Model (GMM) Parameterization: Rather than using a fixed set of "bins" or quantiles, GMAC models the value distribution as a mixture of Gaussians. Because they use GMMs, the Energy Distance has a closed-form analytic solution. This eliminates the need for complex sampling during the loss calculation, making the update step much faster.

Key Insights and Results

Capturing Multimodality

A major advantage of GMAC is its ability to correctly identify when a state has multiple possible outcomes. For example, in a game of Breakout, a state might lead to scoring a point or losing the round. Traditional methods often "blur" these divergent outcomes into a single average, while GMAC successfully preserves the distinct peaks (modes) of the distribution.

Comparison of value distributions in Breakout showing GMAC capturing distinct bimodal peaks compared to IQN

Efficiency and Speed

By using GMMs and analytic solutions, GMAC achieves the benefits of distributional RL with almost the same computational footprint as a standard scalar PPO model. It avoids the heavy "quantile embedding" layers required by models like IQN. In a Breakout benchmark, GMAC’s update step was roughly 2.5x faster than a quantile-based distributional actor-critic (IQAC) and used significantly fewer parameters (0.44M vs 0.52M).

Broad Performance Gains

The researchers evaluated GMAC on a wide variety of standard RL benchmarks, proving its versatility:

  • Atari Performance (Discrete): GMAC outperformed the scalar PPO baseline and other distributional variants (IQN, QR-DQN) in mean human-normalized scores across 57 Arcade Learning Environment (ALE) games.
  • Continuous Control: GMAC proved to be a "drop-in" improvement for continuous tasks in PyBullet, showing particular strength in complex environments like Humanoid.
  • Exploration via Distributional Information: The authors show that the shape of the value distribution itself can be used as an "intrinsic reward." In sparse-reward games like Montezuma’s Revenge, rewarding the agent for reaching states with novel value distributions led to much better exploration than standard methods.
  • Non-Degeneracy: Unlike some quantile methods that can "collapse" or become unstable, the energy distance loss ensured non-degenerative performance across all tested seeds.

Limitations and Future Work

While GMAC provides a robust framework, the authors note a few areas for growth:

  • Hyperparameter Sensitivity: While more stable than previous methods, GMMs still require careful tuning of the number of mixture components.
  • Risk-Sensitive Policies: The paper focuses on using the distribution to improve the accuracy of the mean, but future work could use this distribution to create "risk-averse" or "risk-seeking" agents (e.g., an agent that actively avoids paths with a high probability of a zero-return "catastrophe").
  • Advanced Exploration: The use of Cramér distance as an intrinsic reward is promising but was only briefly explored; more sophisticated "distributional curiosity" remains an open research path.

Summary updated

Abstract

Recent distributional reinforcement learning methods, despite their successes, still contain fundamental problems that can lead to inaccurate representations of value distributions, such as distributional instability, action type restriction, and conflation between samples and statistics. In this paper, we present a novel distributional actor-critic framework, GMAC, to address such problems. Adopting a stochastic policy removes the first two problems, and the conflation in the approximation is alleviated by minimizing the Crame ́r distance between the value distribution and its Bellman target distribution. In addition, GMAC improves data efficiency by generating the Bellman target distribution through the Sample-Replacement algorithm, denoted by SR(λ), which provides a distributional generalization of multi-step policy evaluation algorithms. We empirically show that our method captures the multimodality of value distributions and improves the performance of a conventional actor-critic method with low computational cost in both discrete and continuous action spaces, using Arcade Learning Environment (ALE) and PyBullet environment.
BibTeX
@misc{nam2020distributional,
  author = {Daniel Wontae Nam and Younghoon Kim and Chan Youn Park},
  title = {{A Distributional Perspective on Actor-Critic Framework}},
  year = {2020},
  url = {https://openreview.net/forum?id=jWXBUsWP7N},
  howpublished = {OpenReview},
}
Figure