Abstract
Meta-Reinforcement Learning (meta-RL) aims to develop agents capable of rapid adaptation to novel environments. Traditional approaches often rely on recurrent neural networks (RNNs) to encode environmental context, which can suffer from memory limitations and lack interpretability. This paper presents RAMP (Reinforcement learning Agent using Model Parameters), a framework that utilizes the weights of a dynamics-prediction network as a context vector for a model-free policy. By identifying a sparse subset of environment-specific parameters that capture physical variations, RAMP achieves near-optimal adaptation performance with enhanced transparency.
Introduction
Standard Reinforcement Learning (RL) agents typically struggle when environmental dynamics, such as gravity or friction, deviate from training conditions. Meta-RL addresses this by training agents across a distribution of environments to "learn to learn." A common strategy involves providing the agent with a context vector describing the current environment, often generated by an RNN processing recent history. However, RNNs face challenges with vanishing gradients over long episodes and offer little insight into the agent's internal representation of physics.
Methodology: The RAMP Framework
RAMP decouples the learning process into two phases: dynamics modeling and policy optimization.
Phase 1: Multi-Environment Dynamics Modeling
A neural network is trained to predict the next state $s'$ from the current state and action. The network parameters are partitioned into shared parameters ($\sigma$), representing universal physical laws, and environment-specific parameters ($\omega$), a small subset of weights tuned for each environment. To select $\omega$, RAMP identifies parameters with high gradient variance across environments, indicating sensitivity to environmental shifts.
Phase 2: Policy Learning with Parameter Context
The values of $\omega$ serve as a context vector, concatenated with the state and fed into a model-free RL algorithm (TD3). In a new environment, the agent performs a single probing episode to update $\omega$, allowing the policy to adapt immediately in subsequent episodes.
Evaluation and Results
The framework was evaluated on a sine wave regression benchmark and a vehicle target-reaching simulation with varying braking factors ($k_a$).
Performance Comparison
In the vehicle domain, RAMP demonstrated performance nearly identical to an "Oracle" agent possessing perfect knowledge of the environment's physical constants.
Key results include:
- Accuracy: RAMP achieved a Mean Squared Error of 0.021 in sine wave regression, outperforming MAML (0.037).
- Efficiency: In the vehicle task, RAMP's average reward (-0.1230) closely matched the Oracle (-0.1226), while baselines like MQL lagged significantly.
- Explainability: Probing the $\omega$ parameters allowed for accurate prediction of hidden physical constants, confirming that the model captures meaningful environmental properties.
Conclusion
RAMP offers a transparent and efficient alternative to black-box meta-RL methods by using model parameters as environmental context. While currently focused on variations in dynamics, future work will explore adaptation to changing reward functions and higher-dimensional control tasks.
Summary updated