Fully Parameterized Quantile Function for Distributional Reinforcement Learning

Derek Yang, Li Zhao, Zichuan Lin, Tao Qin, Jiang Bian, Tie-Yan Liu

Abstract

Distributional Reinforcement Learning (RL) differs from traditional RL in that, rather than the expectation of total returns, it estimates distributions and has achieved state-of-the-art performance on Atari Games. The key challenge in practical distributional RL algorithms lies in how to parameterize estimated distributions so as to better approximate the true continuous distribution. Existing distributional RL algorithms parameterize either the probability side or the return value side of the distribution function, leaving the other side uniformly fixed as in C51, QR-DQN or randomly sampled as in IQN. In this paper, we propose fully parameterized quantile function that parameterizes both the quantile fraction axis (i.e., the x-axis) and the value axis (i.e., y-axis) for distributional RL. Our algorithm contains a fraction proposal network that generates a discrete set of quantile fractions and a quantile value network that gives corresponding quantile values. The two networks are jointly trained to find the best approximation of the true distribution. Experiments on 55 Atari Games show that our algorithm significantly outperforms existing distributional RL algorithms and creates a new record for the Atari Learning Environment for non-distributed agents.

Introduction

In standard Reinforcement Learning (RL), agents typically aim to maximize the expected total return. Distributional RL advances this by estimating the entire probability distribution of potential returns, allowing agents to better account for environmental randomness and policy uncertainty.

The primary hurdle in distributional RL is parameterization: representing a continuous probability distribution with a finite set of parameters. Previous methods have been "half-fixed":

  • C51 (Categorical DQN) fixes return values and learns probabilities.
  • QR-DQN fixes probabilities (quantile fractions) and learns return values.
  • IQN (Implicit Quantile Network) randomly samples probabilities.

Because return distributions are rarely uniform, these fixed or random approaches are inefficient, often wasting parameters on flat regions of the distribution while missing critical details in high-variance areas.

Methodology: Fully Parameterized Quantile Function (FQF)

FQF introduces a "fully parameterized" approach by using two interconnected neural networks to represent the quantile function:

  1. Fraction Proposal Network (FPN): This network takes the state embedding and adaptively "proposes" a set of $N$ optimal quantile fractions ($\tau$). Unlike previous methods, these fractions are state-dependent, allowing the agent to "zoom in" on complex parts of the distribution. The FPN is trained to minimize the 1-Wasserstein distance between the estimated and true distribution.
  2. Quantile Value Network (QVN): This network maps the proposed fractions to their corresponding quantile values (return values).
Comparison between optimal (a) and random (b) quantile fractions for approximating the quantile function.

To ensure the fractions are valid (ordered between 0 and 1), the authors utilize a cumulated softmax layer. By making the quantile fractions learnable, the agent can adapt its resolution based on the current game state, such as focusing more precisely on the distribution when an enemy appears in Kung Fu Master.

Experimental Results

The algorithm was evaluated on the Arcade Learning Environment (ALE) across 55 Atari 2600 games. FQF demonstrated significant performance gains over previous state-of-the-art methods.

Performance comparison across 55 Atari games. FQF achieves a new state-of-the-art mean human-normalized score of 1426%.

Key Findings:

  • New State-of-the-Art: FQF achieved a 1426% mean human-normalized score, substantially outperforming IQN (1112%) and the massive "Rainbow" agent (1213%).
  • Superior Robustness: FQF reached human-level performance on 44 out of 55 games, setting a new record for non-distributed agents.
  • Learning Efficiency: In several games, FQF converged to high scores faster than IQN, as its self-adjusting fractions allowed for more efficient learning updates in critical regions of the return distribution.

Discussion and Limitations

While FQF sets a new performance benchmark, it introduces a computational overhead of approximately 20% slower training per iteration due to the additional Fraction Proposal Network. Furthermore, while the algorithm performs exceptionally in practice, theoretical convergence for the fraction proposal network when combined with non-linear function approximation remains an open research question.

Future work may involve enforcing the non-decreasing property of quantile functions as a prior and extending this adaptive parameterization to continuous control tasks in robotics.

Summary updated

Abstract

Distributional Reinforcement Learning (RL) differs from traditional RL in that, rather than the expectation of total returns, it estimates distributions and has achieved state-of-the-art performance on Atari Games. The key challenge in practical distributional RL algorithms lies in how to parameterize estimated distributions so as to better approximate the true continuous distribution. Existing distributional RL algorithms parameterize either the probability side or the return value side of the distribution function, leaving the other side uniformly fixed as in C51, QR-DQN or randomly sampled as in IQN. In this paper, we propose fully parameterized quantile function that parameterizes both the quantile fraction axis (i.e., the x-axis) and the value axis (i.e., y-axis) for distributional RL. Our algorithm contains a fraction proposal network that generates a discrete set of quantile fractions and a quantile value network that gives corresponding quantile values. The two networks are jointly trained to find the best approximation of the true distribution. Experiments on 55 Atari Games show that our algorithm significantly outperforms existing distributional RL algorithms and creates a new record for the Atari Learning Environment for non-distributed agents.
BibTeX
@inproceedings{yang2019fully,
  author = {Derek Yang and Li Zhao and Zichuan Lin and Tao Qin and Jiang Bian and Tie-Yan Liu},
  title = {{Fully Parameterized Quantile Function for Distributional Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=Wr1vGYKTFRF},
}

Loading related content...

Figure