Meta-Reinforcement Learning Using Model Parameters

Gabriel Hartmann, Amos AzariaarXiv 2022 · DOI · arXiv

Abstract

Meta-Reinforcement Learning (meta-RL) aims to develop agents capable of rapid adaptation to novel environments. Traditional approaches often rely on recurrent neural networks (RNNs) to encode environmental context, which can suffer from memory limitations and lack interpretability. This paper presents RAMP (Reinforcement learning Agent using Model Parameters), a framework that utilizes the weights of a dynamics-prediction network as a context vector for a model-free policy. By identifying a sparse subset of environment-specific parameters that capture physical variations, RAMP achieves near-optimal adaptation performance with enhanced transparency.

Introduction

Standard Reinforcement Learning (RL) agents typically struggle when environmental dynamics, such as gravity or friction, deviate from training conditions. Meta-RL addresses this by training agents across a distribution of environments to "learn to learn." A common strategy involves providing the agent with a context vector describing the current environment, often generated by an RNN processing recent history. However, RNNs face challenges with vanishing gradients over long episodes and offer little insight into the agent's internal representation of physics.

Methodology: The RAMP Framework

RAMP decouples the learning process into two phases: dynamics modeling and policy optimization.

Phase 1: Multi-Environment Dynamics Modeling

A neural network is trained to predict the next state $s'$ from the current state and action. The network parameters are partitioned into shared parameters ($\sigma$), representing universal physical laws, and environment-specific parameters ($\omega$), a small subset of weights tuned for each environment. To select $\omega$, RAMP identifies parameters with high gradient variance across environments, indicating sensitivity to environmental shifts.

Environment-specific parameters $\omega$ show distinct values for different physical settings, acting as a "DNA" for the environment.

Phase 2: Policy Learning with Parameter Context

The values of $\omega$ serve as a context vector, concatenated with the state and fed into a model-free RL algorithm (TD3). In a new environment, the agent performs a single probing episode to update $\omega$, allowing the policy to adapt immediately in subsequent episodes.

Evaluation and Results

The framework was evaluated on a sine wave regression benchmark and a vehicle target-reaching simulation with varying braking factors ($k_a$).

Performance Comparison

In the vehicle domain, RAMP demonstrated performance nearly identical to an "Oracle" agent possessing perfect knowledge of the environment's physical constants.

Reward curves show RAMP achieving near-optimal performance, significantly outperforming Basic RL and Meta-Q-Learning (MQL).

Key results include:

  • Accuracy: RAMP achieved a Mean Squared Error of 0.021 in sine wave regression, outperforming MAML (0.037).
  • Efficiency: In the vehicle task, RAMP's average reward (-0.1230) closely matched the Oracle (-0.1226), while baselines like MQL lagged significantly.
  • Explainability: Probing the $\omega$ parameters allowed for accurate prediction of hidden physical constants, confirming that the model captures meaningful environmental properties.

Conclusion

RAMP offers a transparent and efficient alternative to black-box meta-RL methods by using model parameters as environmental context. While currently focused on variations in dynamics, future work will explore adaptation to changing reward functions and higher-dimensional control tasks.

Summary updated

Abstract

In meta-reinforcement learning, an agent is trained in multiple different environments and attempts to learn a meta-policy that can efficiently adapt to a new environment. This paper presents RAMP, a Reinforcement learning Agent using Model Parameters that utilizes the idea that a neural network trained to predict environment dynamics encapsulates the environment information. RAMP is constructed in two phases: in the first phase, a multi-environment parameterized dynamic model is learned. In the second phase, the model parameters of the dynamic model are used as context for the multi-environment policy of the model-free reinforcement learning agent.
BibTeX
@misc{hartmann2022meta,
  author = {Gabriel Hartmann and Amos Azaria},
  title = {{Meta-Reinforcement Learning Using Model Parameters}},
  year = {2022},
  eprint = {2210.15515},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.15515},
}
Figure