Distributional Meta-Gradient Reinforcement Learning

Haiyan Yin, Shuicheng Yan, Zhongwen XuICLR 2023 · OpenReview

TL;DR

Researchers have combined meta-gradient reinforcement learning (RL) with distributional RL to create DrMG (Distributional Meta-Gradient). By learning to adapt the entire distribution of future rewards rather than just a single average value, the agent gains a much richer signal for self-tuning, leading to state-of-the-art performance on the Atari 2600 benchmark.

The Limitation of Expected Returns in Meta-RL

Reinforcement learning agents typically try to maximize the "expected return"—the average amount of reward they expect to get in the future. Meta-gradient RL (MGRL) takes this a step further by allowing the agent to "learn how to learn" by automatically tuning its own hyperparameters (like the discount factor $\gamma$) using gradients.

However, traditional MGRL has a major limitation: it treats rewards as scalars (single numbers). This "averaging out" of the future discards vital information about risk, uncertainty, and the variety of possible outcomes. In complex environments where rewards are sparse or noisy, a single scalar doesn't provide enough "feedback" for the meta-learning process to work efficiently. The authors argue that by moving from scalars to value distributions, the agent can access a denser, more informative signal to optimize its learning objective.

The DrMG Approach

The authors introduce DrMG, which integrates Quantile Distributional RL into a meta-gradient framework.

  1. Distributional $\lambda$-Return: Instead of a standard scalar return, they formulate a distributional version of the TD($\lambda$) return. This represents the return as a set of "quantiles" (slices of a probability distribution).
  2. Vectorized Meta-Parameters: In previous work, hyperparameters like the discount factor ($\gamma$) were single numbers. In DrMG, these can be vectors. This means the agent can learn different discount factors or bootstrapping weights for different parts of the reward distribution (e.g., being more short-sighted about rare high-reward events while being long-sighted about common small rewards).
  3. Bi-level Optimization: The algorithm uses a "white-box" meta-learning approach.
  4. Inner Loop: The agent updates its neural network weights to minimize a distributional loss.
  5. Outer Loop: The agent evaluates how that update performed on a new batch of data and adjusts its meta-parameters (like $\gamma, \lambda$, and loss weights) to improve future performance.
  6. Architecture: The system is built on top of IMPALA, a high-throughput distributed actor-critic framework, using a ResNet backbone.

Experimental Setup

The researchers evaluated DrMG across two primary settings:

  • Two-Colors Grid-World: A custom toy environment designed to test adaptation. The rewards for different "keys" are toggled periodically, forcing the agent to quickly adapt its internal model of the return distribution.
  • Atari 2600 (57 games): The standard large-scale benchmark for RL. The authors evaluated on the "200M frame" regime, which is the rigorous standard for measuring sample efficiency and final performance in the field.

Key Results

Adaptability in Non-Stationary Environments In the toy grid-world, DrMG adapted to reward changes much faster than scalar-based agents. When the environment "toggled" its rules, DrMG reached higher peaks of performance and recovered more quickly, demonstrating the value of distributional signals for rapid adaptation.

Grid-world environment and adaptation performance

State-of-the-Art Atari Performance DrMG achieved a median Human Normalized Score (HNS) of 379% across 57 Atari games. This significantly outperforms the standard meta-gradient baseline (MG), which scored 287%. It reached the highest median HNS reported in the meta-gradient RL literature for the 200M frame Atari benchmark.

Median Human Normalized Score across 57 Atari games

Ablation Success The researchers showed that using vectorized hyperparameters (giving each quantile its own $\gamma$ and $\lambda$) was superior to using a single scalar hyperparameter for the whole distribution. Furthermore, DrMG is more computationally efficient than some previous meta-RL attempts (like STACX) because it avoids adding multiple auxiliary "heads" to the neural network, instead focusing on the inherent structure of the distribution.

Limitations and Future Work

  • On-Policy Focus: The current work is primarily designed for on-policy actor-critic methods (like IMPALA). The authors suggest that extending this to off-policy settings (like pure DQN variants) or offline RL is a logical next step.
  • Continuous Control: The experiments were limited to discrete action spaces (Atari). Testing DrMG in continuous control environments (like MuJoCo or robotics simulations) remains for future research.
  • Complexity of Tuning: While the agent tunes its own $\gamma$ and $\lambda$, the meta-learning process itself introduces new "meta-hyperparameters" (like the meta-learning rate) that still require manual setting.

Summary updated

Abstract

Meta-gradient reinforcement learning (RL) algorithms have substantially boosted the performance of RL agents by learning an adaptive return. All the existing algorithms adhere to the same reward learning principle, where the adaptive return is simply formulated in the form of expected cumulative rewards, upon which the policy and critic update rules are specified under well-adopted distance metrics. In this paper, we present a novel algorithm that builds on the success of meta-gradient RL algorithms and effectively improves such algorithms by following a simple recipe, i.e., going beyond the expected return to formulate and learn the return in a more expressive form, value distributions. To this end, we first formulate a distributional return that could effectively capture bootstrapping and discounting behaviors over distributions, to form an informative distributional return target in value update. Then we derive an efficient meta update rule to learn the adaptive distributional return with meta-gradients. For empirical evaluation, we first present an illustrative example on a toy two-color grid-world domain, which validates the benefit of learning distributional return over expectation; then we conduct extensive comparisons on a large-scale RL benchmark Atari 2600, where we confirm that our proposed method with distributional return works seamlessly well with the actor-critic framework and leads to state-of-the-art median human normalized score among meta-gradient RL literature.
BibTeX
@inproceedings{yin2023distributional,
  author = {Haiyan Yin and Shuicheng YAN and Zhongwen Xu},
  title = {{Distributional Meta-Gradient Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=LGkmUauBUL},
}
Figure