Introduction
In model-based reinforcement learning, a primary goal is to learn a forward dynamics model that accurately predicts future states. However, real-world environments are non-static; a robot's behavior changes based on its payload, or a vehicle's performance varies with terrain. While meta-learning allows models to adapt to these variations using a "context set" of observations, most existing methods rely on random data collection to build this context. This paper introduces a method for informed calibration, enabling an agent to actively collect the most informative data to identify hidden system properties (like mass or friction) in just a few steps.
The Proposed Framework
The authors propose a two-stage framework: first, training a context-conditional dynamics model on a variety of environments, and second, using that model to perform active calibration in a new, unknown environment.
Context-Conditional Dynamics Model
The core of the system is built on the Neural Processes (NP) framework. The model learns a distribution over possible dynamics by mapping observed transitions—state, action, and next state—into a latent context variable $\beta$.
To ensure stable and reliable inference, the authors implement a "strictly decreasing variance" constraint. By using non-negative weights and Softplus activations in the variance regressor, the architecture is forced to be monotonic. This bakes the logic of "more data equals less uncertainty" directly into the model, mimicking ideal Bayesian inference and making optimization more robust.
Informed Calibration via Expected Information Gain (EIG)
To decide which actions to take during the calibration phase, the agent maximizes Expected Information Gain (EIG). Instead of seeking a traditional task reward, the agent treats uncertainty as a negative reward. It "imagines" potential action sequences and selects the one predicted to result in a context set that most significantly reduces the entropy of the latent variable $\beta$.
This calibration is implemented via Model Predictive Control (MPC), where the agent replans its action sequence at every timestep. This allows the agent to incorporate new information as it arrives, dynamically adjusting its exploration strategy to target the most ambiguous aspects of the system.
Experimental Evaluation
The researchers evaluated their approach on three simulated environments: a non-linear toy system, a modified OpenAI Gym Pendulum (with varying mass and torque), and a MountainCar environment with 60,000 unique terrain profiles.
Key Insights
- Knowledge as a Reward: In the Pendulum task, the agent naturally performs a "swing-up" maneuver without being told to. It realizes it cannot reduce uncertainty about the upper quadrants of the swing without actually visiting them.
- Efficiency and Accuracy: Across all tasks, informed calibration sequences resulted in significantly lower mean squared error (MSE) in dynamics prediction compared to random actions. In the toy problem, the model often reached minimum uncertainty in just one or two transitions.
- Downstream Performance: A model calibrated using the MPC-EIG scheme allowed an agent to perform a swing-up task with performance nearly identical to an agent using ground-truth dynamics, whereas models calibrated with random data failed significantly.
Conclusion and Future Work
This work demonstrates that active, information-theoretic exploration is far more efficient than random data collection for adapting dynamics models. By treating the identification of a system as an optimal experimental design problem, agents can learn to navigate new environments with minimal downtime. Future research directions include extending this framework to Partially Observable Markov Decision Processes (POMDPs) and investigating how these models handle "out-of-distribution" systems that fall outside the family of environments seen during training.
Summary updated