Traditional Reinforcement Learning (RL) typically focuses on the expected return—the average reward an agent expects to receive over time. Distributional RL, however, takes a more comprehensive approach by modeling the entire probability distribution of returns. This is a powerful paradigm shift because it captures the intrinsic randomness of both the environment and the agent's policy, leading to more robust learning.
The core challenge in practical distributional RL is parameterization: how do we accurately represent a continuous probability distribution using a finite set of numbers? Previous state-of-the-art methods have only "half-parameterized" this problem:
- C51 fixes the return values (the x-axis) and learns their corresponding probabilities.
- QR-DQN fixes the probabilities (the y-axis, using uniform "bins") and learns the return values.
- IQN randomly samples probabilities to approximate the full function.
Because these methods do not optimize where the "probes" (quantile fractions) should be placed, they can be inefficient. If an agent only has a limited number of quantiles (e.g., 32) to represent a complex distribution, the exact placement of those quantiles matters immensely for accuracy.
The Fully Parameterized Approach
To address this limitation, researchers introduced the Fully Parameterized Quantile Function (FQF). FQF is a distributional RL algorithm that learns to adaptively pick the best "slices" (quantile fractions) of a return distribution while simultaneously learning their values.
By parameterizing the entire quantile function rather than just parts of it, the agent learns to "pay more attention" to the parts of the distribution that are most complex or important.
The FQF architecture consists of two jointly trained neural networks:
- Fraction Proposal Network (FPN): This network takes the state embedding and generates a set of $N$ quantile fractions ($\tau$). To ensure these fractions are valid (sorted between 0 and 1), the network outputs a set of values that pass through a softmax layer and are then cumulatively summed.
- Quantile Value Network (QVN): Similar to the network used in IQN, this component maps the proposed fractions to their corresponding return values ($\theta$).
The Wasserstein Gradient Trick: A major innovation in FQF is how it trains the FPN. Computing the full 1-Wasserstein distance between distributions is computationally expensive. The authors derived a specific gradient for the 1-Wasserstein distance with respect to the quantile fractions. This mathematical trick allows the FPN to move the fractions to locations that minimize the distortion between the approximated distribution and the true distribution, making the optimization practical for deep RL.
While IQN relies on the law of large numbers—sampling many random fractions over time to approximate the distribution—FQF uses a small number of optimal fractions to achieve better results with significantly less variance.
Key Results on Atari-57
The algorithm was evaluated on the Arcade Learning Environment (ALE), specifically a standard suite of 55 Atari 2600 games. FQF was compared against major baselines including DQN, QR-DQN, IQN, and the non-distributed version of Rainbow.
FQF significantly outperformed existing state-of-the-art distributional RL algorithms:
- Unprecedented Performance: On the 55 Atari games, FQF achieved a mean human-normalized score of 1426% and a median score of 272%. For comparison, the previous leader (IQN) scored 1112% mean and 218% median.
- New Record: FQF set a new record for non-distributed agents (agents that do not use massive parallelization across many CPUs/GPUs) on the ALE benchmark.
- Faster Convergence: In many games, such as Centipede, FQF converged to high scores much faster than IQN. The self-adjusting fractions likely provided a more stable and accurate signal for the value network during training.
Limitations and Future Work
Despite its impressive performance, FQF comes with a few trade-offs:
- Computational Cost: FQF is roughly 20% slower than IQN in terms of wall-clock training time per step. This is due to the overhead of running the additional Fraction Proposal Network and calculating its specific gradients.
- Scaling: As the number of quantiles ($N$) increases, the computational burden of FQF grows more significantly than IQN's.
- Theoretical Convergence: While the authors provide strong empirical proof of success, guaranteeing convergence for the fraction proposal network in a deep RL setting—where the target distribution is constantly shifting due to Bellman updates—remains an open theoretical question.
- Exploration: Like most distributional methods, FQF still struggles with "hard exploration" games like Montezuma's Revenge. This suggests that better distribution modeling alone is not enough to solve complex logic puzzles, highlighting an area for future algorithmic integration.
Summary updated