Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning

Younggyo Seo, Kimin Lee, Ignasi Clavera Gilaberte, Thanard Kurutach, Jinwoo Shin, Pieter Abbeel

TL;DR

T-MCL is a model-based reinforcement learning algorithm designed to handle environments with changing or unpredictable physics. By training a multi-headed dynamics model where each head specializes in a specific "mode" of physics (e.g., different friction levels or robot damage), and using an adaptive planning mechanism to pick the best head on the fly, it achieves state-of-the-art zero-shot generalization to unseen environments.

What Problem Does This Paper Address?

Model-based Reinforcement Learning (RL) is highly efficient because the agent learns a "world model" to simulate the consequences of its actions. However, these models are notoriously brittle. If a robot trained on flat ground suddenly encounters sand, or if one of its motors loses power, the transition dynamics change.

Standard models struggle because these changes often create a multi-modal distribution. For example, an "average" model might try to predict a state that is halfway between "walking with four legs" and "walking with three legs," resulting in a prediction that is accurate for neither.

Multi-modal dynamics in the CrippledAnt environment.

The authors aim to create a model that can represent these distinct modes and adapt to them without needing explicit labels for which environment it is currently in.

Methods

The authors introduce Trajectory-wise Multiple Choice Learning (T-MCL), which tackles this multi-modal challenge using three core components:

Overview of the T-MCL architecture.
  1. Multi-headed Dynamics Model: Instead of a single output, the model has multiple "heads" sharing a common backbone network.
  2. Trajectory-wise Oracle Loss: This is the "secret sauce" for specialization. During training, the algorithm looks at a segment of a trajectory (a sequence of states and actions). It calculates the prediction error for every head but only updates the parameters of the most accurate head for that specific segment. This encourages each head to specialize in a specific cluster of dynamics (e.g., Head 1 learns "high friction," Head 2 learns "low mass") in an unsupervised way.
  3. Context Learning: A context encoder processes a window of past transitions to produce a latent vector. This vector is fed into the dynamics heads, providing a "hint" about the current environment's properties.
  4. Adaptive Planning: At test time, the agent doesn't know which head to use. It performs "online adaptation" by looking at its most recent transitions, identifying which head would have predicted them most accurately, and then using that specific head for its Model Predictive Control (MPC) planning.

Datasets

The researchers evaluated T-MCL on several classic and complex control tasks:

  • OpenAI Gym: CartPoleSwingUp and Pendulum.
  • MuJoCo Physics Engine: Hopper, SlimHumanoid, HalfCheetah, and CrippledAnt.

To test generalization, they created multi-modal environments by varying physical parameters (mass, length) or "crippling" specific robot legs. They trained the agents on a discrete set of these parameters and tested them on unseen parameters outside the training range (zero-shot generalization).

Key Insights and Ideas

  • Specialization via Competition: The "Oracle Loss" acts as an unsupervised clustering mechanism. By only rewarding the "winner" of a prediction task, the model naturally partitions the environment space without needing human-provided labels for different dynamics.
  • Trajectory vs. Step: A key finding was that calculating the oracle loss over a trajectory segment (multiple steps) is much more stable than doing it step-by-step. Step-wise updates often fail to produce distinct, specialized heads.
  • Planning with the Best, Not the Average: In multi-modal settings, averaging the predictions of an ensemble (a common RL technique) leads to "blurry" and inaccurate results. Selecting the single best-performing head for the current context is significantly more effective for planning.

Key Results

  • Superior Generalization: T-MCL significantly outperformed existing model-based baselines like PETS and CaDM, as well as meta-RL methods like PEARL.
  • Massive Gains on CrippledAnt: In the CrippledAnt environment (where the robot must adapt to different legs being disabled), T-MCL achieved a 3.5x higher average return than the previous state-of-the-art (CaDM).
  • Latent Space Separation: Visualizations of the context encoder's latent space showed that the model successfully learned to separate different environment dynamics (e.g., different masses) into distinct clusters, even though it was never told which mass it was experiencing.

Limitations and Future Work

  • Hyperparameter Sensitivity: The performance is sensitive to the number of heads and the horizon of the trajectory segments. If the number of heads is too high, the model might overfit or split similar dynamics unnecessarily.
  • Known Reward Function: Like many model-based planning papers, this work assumes the reward function is known and doesn't change, even if the dynamics do. In some real-world scenarios, the reward (the "goal") might also be partially hidden or change alongside the physics.
  • Computational Overhead: While more sample-efficient in terms of environment steps, maintaining and planning with multiple heads adds computational complexity during training and inference.

Summary updated

Abstract

Model-based reinforcement learning (RL) has shown great potential in various control tasks in terms of both sample-efficiency and final performance. However, learning a generalizable dynamics model robust to changes in dynamics remains a challenge since the target transition dynamics follow a multi-modal distribution. In this paper, we present a new model-based RL algorithm, coined trajectory-wise multiple choice learning, that learns a multi-headed dynamics model for dynamics generalization. The main idea is updating the most accurate prediction head to specialize each head in certain environments with similar dynamics, i.e., clustering environments. Moreover, we incorporate context learning, which encodes dynamics-specific information from past experiences into the context latent vector, enabling the model to perform online adaptation to unseen environments. Finally, to utilize the specialized prediction heads more effectively, we propose an adaptive planning method, which selects the most accurate prediction head over a recent experience. Our method exhibits superior zero-shot generalization performance across a variety of control tasks, compared to state-of-the-art RL methods. Source code and videos are available at https://sites.google.com/view/trajectory-mcl.
BibTeX
@inproceedings{seo2020trajectory,
  author = {Younggyo Seo and Kimin Lee and Ignasi Clavera Gilaberte and Thanard Kurutach and Jinwoo Shin and Pieter Abbeel},
  title = {{Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=BG4vr0SGfq},
}
Figure