DynaMITE-RL: A Dynamic Model for Improved Temporal Meta-Reinforcement Learning

Anthony Liang, Guy Tennenholtz, Chih-wei Hsu, Yinlam Chow, Erdem Bıyık, Craig Boutilier

TL;DR

DynaMITE-RL is a new meta-reinforcement learning framework designed for environments where hidden factors—like a user’s changing mood or a robot’s mechanical wear—evolve over time. By explicitly modeling "sessions" of stability and using novel consistency and masking techniques, it significantly outperforms existing methods in both sample efficiency and the ability to adapt to shifting tasks mid-episode.

What Problem Does This Paper Address?

Standard Reinforcement Learning (RL) typically assumes a static environment. Meta-RL attempts to bridge the gap by learning to adapt to different tasks, but most state-of-the-art methods (like VariBAD) assume that once a task is set, it remains fixed for the entire duration of an episode.

In the real world, this assumption fails. A recommender system deals with users whose preferences shift; a manufacturing robot experiences gradual hardware degradation. The authors formalize this challenge as a Dynamic Latent Contextual MDP (DLCMDP). In this setting, the environment is governed by a hidden "latent context" that remains fixed for a period (a "session") before transitioning to a new state. Existing meta-RL agents often get "stuck" or confused when these transitions occur because they try to find a single explanation for a trajectory that actually contains multiple different tasks.

Methods

The core of DynaMITE-RL is its ability to exploit the temporal structure of sessions. The authors introduce three key modifications to the standard variational meta-RL pipeline:

  1. Consistency of Latent Information: Within a session, the latent context is fixed. The authors introduce a consistency loss that encourages the agent’s belief to become progressively more certain as it gathers more data, preventing erratic jumps in its internal representation of the task.
  2. Session Reconstruction Masking: Traditional methods try to reconstruct the entire episode trajectory from a single latent belief. DynaMITE-RL instead uses "masking" to ensure the model only attempts to reconstruct transitions that share the same latent context. This prevents the model from being biased by past or future sessions with different dynamics.
  3. Prior Latent Conditioning: Unlike models that treat each episode as a blank slate, DynaMITE-RL conditions its belief for a new session on the final belief of the previous one. This allows the agent to learn the underlying "meta-rules" of how the environment changes (e.g., learning that a goal usually moves clockwise).

The architecture, shown in the figure below, uses a recurrent inference network to predict both the latent context and the probability that a session has ended.

Figure 3: Pseudo-code (online RL training) and model architecture of DynaMITE-RL.

Datasets

The researchers evaluated DynaMITE-RL on a diverse suite of benchmarks modified to include temporal shifts:

  • Gridworld Alternate: A 5x5 navigation task where the rewarding goal location swaps between two spots according to a transition matrix.
  • MuJoCo Continuous Control: Standard locomotion tasks (Reacher, HalfCheetah) modified so that target velocities, directions, or external wind forces change mid-episode.
  • Assistive Itch Scratching: A complex human-in-the-loop simulation from Assistive-Gym where a robot arm must find and scratch an itch that moves along a person's arm.

Key Insights and Ideas

  • The "Session" Inductive Bias: The most powerful idea here is that non-stationarity isn't usually chaotic; it's "piecewise stationary." By building this assumption into the loss function, the agent learns much faster.
  • Reconstruction is a Double-Edged Sword: In static meta-RL, reconstructing the whole episode helps the agent understand the task. In dynamic settings, it's a trap. Trying to explain two different tasks with one latent vector leads to a "suboptimal average" policy. Masking is the surgical fix for this.
  • Adapting to the Change: As seen in Figure 2, while baselines like VariBAD fail to adapt once a goal moves, DynaMITE-RL correctly infers the transition and consistently reaches the new rewarding location.
DynaMITE-RL vs VariBAD in a changing Gridworld

Key Results

DynaMITE-RL demonstrated substantial improvements over baselines like RL², VariBAD, and SecBAD:

  • Higher Returns: In the Reacher-Circle task, DynaMITE-RL achieved an average return of -8.4, compared to -102.4 for VariBAD.
  • Superior Adaptation: In the ScratchItch-Jaco task, it achieved a return of 231.2, nearly tripling the performance of the best baseline (82.5).
  • Robustness: The model proved effective in both online (learning while interacting) and offline (learning from a fixed dataset) RL settings, showing that these architectural improvements are fundamental to belief modeling, not just exploration.

Limitations and Future Work

The authors note that their current model assumes Markovian latent dynamics, meaning the next hidden state depends only on the current one. Real-world scenarios might involve more complex, non-Markovian dependencies.

Future work aims to explore hierarchical latent contexts (where hidden factors change at different timescales) and applying the framework to large-scale real-world applications like autonomous driving and multi-agent coordination.

Summary updated

Abstract

We introduce DynaMITE-RL, a meta-reinforcement learning (meta-RL) approach to approximate inference in environments where the latent state evolves at varying rates. We model episode sessions - parts of the episode where the latent state is fixed - and propose three key modifications to existing meta-RL methods: consistency of latent information within sessions, session masking, and prior latent conditioning. We demonstrate the importance of these modifications in various domains, ranging from discrete Gridworld environments to continuous-control and simulated robot assistive tasks, demonstrating that DynaMITE-RL significantly outperforms state-of-the-art baselines in sample efficiency and inference returns.
BibTeX
@inproceedings{liang2024dynamite,
  author = {Anthony Liang and Guy Tennenholtz and Chih-wei Hsu and Yinlam Chow and Erdem Bıyık and Craig Boutilier},
  title = {{DynaMITE-RL: A Dynamic Model for Improved Temporal Meta-Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=OPrPegYIZo},
}
Figure