Explore the Context: Optimal Data Collection for Context-Conditional Dynamics Models

Jan Achterhold, Joerg Stueckler

Introduction

In model-based reinforcement learning, a primary goal is to learn a forward dynamics model that accurately predicts future states. However, real-world environments are non-static; a robot's behavior changes based on its payload, or a vehicle's performance varies with terrain. While meta-learning allows models to adapt to these variations using a "context set" of observations, most existing methods rely on random data collection to build this context. This paper introduces a method for informed calibration, enabling an agent to actively collect the most informative data to identify hidden system properties (like mass or friction) in just a few steps.

The Proposed Framework

The authors propose a two-stage framework: first, training a context-conditional dynamics model on a variety of environments, and second, using that model to perform active calibration in a new, unknown environment.

The framework consists of a training phase on diverse datasets and an MPC-based calibration loop in the test environment.

Context-Conditional Dynamics Model

The core of the system is built on the Neural Processes (NP) framework. The model learns a distribution over possible dynamics by mapping observed transitions—state, action, and next state—into a latent context variable $\beta$.

Graphical model showing the dependencies between context $C^\alpha$, latent variable $\beta$, and the sequence of states $x$ and actions $u$.

To ensure stable and reliable inference, the authors implement a "strictly decreasing variance" constraint. By using non-negative weights and Softplus activations in the variance regressor, the architecture is forced to be monotonic. This bakes the logic of "more data equals less uncertainty" directly into the model, mimicking ideal Bayesian inference and making optimization more robust.

Informed Calibration via Expected Information Gain (EIG)

To decide which actions to take during the calibration phase, the agent maximizes Expected Information Gain (EIG). Instead of seeking a traditional task reward, the agent treats uncertainty as a negative reward. It "imagines" potential action sequences and selects the one predicted to result in a context set that most significantly reduces the entropy of the latent variable $\beta$.

This calibration is implemented via Model Predictive Control (MPC), where the agent replans its action sequence at every timestep. This allows the agent to incorporate new information as it arrives, dynamically adjusting its exploration strategy to target the most ambiguous aspects of the system.

Experimental Evaluation

The researchers evaluated their approach on three simulated environments: a non-linear toy system, a modified OpenAI Gym Pendulum (with varying mass and torque), and a MountainCar environment with 60,000 unique terrain profiles.

Key Insights

  • Knowledge as a Reward: In the Pendulum task, the agent naturally performs a "swing-up" maneuver without being told to. It realizes it cannot reduce uncertainty about the upper quadrants of the swing without actually visiting them.
  • Efficiency and Accuracy: Across all tasks, informed calibration sequences resulted in significantly lower mean squared error (MSE) in dynamics prediction compared to random actions. In the toy problem, the model often reached minimum uncertainty in just one or two transitions.
  • Downstream Performance: A model calibrated using the MPC-EIG scheme allowed an agent to perform a swing-up task with performance nearly identical to an agent using ground-truth dynamics, whereas models calibrated with random data failed significantly.

Conclusion and Future Work

This work demonstrates that active, information-theoretic exploration is far more efficient than random data collection for adapting dynamics models. By treating the identification of a system as an optimal experimental design problem, agents can learn to navigate new environments with minimal downtime. Future research directions include extending this framework to Partially Observable Markov Decision Processes (POMDPs) and investigating how these models handle "out-of-distribution" systems that fall outside the family of environments seen during training.

Summary updated

Abstract

In this paper, we learn dynamics models for parametrized families of dynamical systems with varying properties. The dynamics models are formulated as stochastic processes conditioned on a latent context variable which is inferred from observed transitions of the respective system. The probabilistic formulation allows us to compute an action sequence which, for a limited number of environment interactions, optimally explores the given system within the parametrized family. This is achieved by steering the system through transitions being most informative for the context variable. We demonstrate the effectiveness of our method for exploration on a non-linear toy-problem and two well-known reinforcement learning environments.
BibTeX
@inproceedings{achterhold2021explore,
  author = {Jan Achterhold and Joerg Stueckler},
  title = {{Explore the Context: Optimal Data Collection for Context-Conditional Dynamics Models}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2021},
}
Figure