Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning

Younggyo Seo, Kimin Lee, Ignasi Clavera, Thanard Kurutach, Jinwoo Shin, Pieter Abbeel

Abstract

Model-based reinforcement learning (RL) has shown great potential in various control tasks in terms of both sample-efficiency and final performance. However, learning a generalizable dynamics model robust to changes in dynamics remains a challenge since the target transition dynamics follow a multi-modal distribution. In this paper, we present a new model-based RL algorithm, coined trajectory-wise multiple choice learning, that learns a multi-headed dynamics model for dynamics generalization. The main idea is updating the most accurate prediction head to specialize each head in certain environments with similar dynamics, i.e., clustering environments. Moreover, we incorporate context learning, which encodes dynamics-specific information from past experiences into the context latent vector, enabling the model to perform online adaptation to unseen environments. Finally, to utilize the specialized prediction heads more effectively, we propose an adaptive planning method, which selects the most accurate prediction head over a recent experience. Our method exhibits superior zero-shot generalization performance across a variety of control tasks, compared to state-of-the-art RL methods.

Introduction: The Challenge of Multi-modal Dynamics

Model-based Reinforcement Learning (RL) is highly valued for its sample efficiency, as it learns a predictive "world model" of environment physics to plan future actions. However, these models are often brittle. When environment dynamics change—such as variations in mass, friction, or hardware malfunctions—standard models fail because they cannot account for the resulting multi-modality in transition dynamics.

Multi-modality in transition dynamics for a crippled ant.

The core issue is that transition dynamics in these scenarios follow a multi-modal distribution. For instance, in the "CrippledAnt" task, the same action leads to vastly different state transitions depending on which leg is disabled. Standard neural networks tend to average these modes, resulting in poor predictions for all possible configurations.

Methodology: Trajectory-wise Multiple Choice Learning (T-MCL)

To address this, T-MCL introduces a robust world model composed of three integrated components:

1. Multi-headed Dynamics Model with Trajectory-wise Oracle Loss

Instead of a single prediction head, the model utilizes multiple heads ($H$). During training, a Trajectory-wise Oracle Loss is applied: the algorithm calculates prediction errors for each head over a trajectory segment. Only the most accurate head for that segment is updated. This "winner-takes-all" mechanism encourages unsupervised specialization, where different heads learn to represent distinct physical "clusters" (e.g., one head for high friction, another for low friction) without requiring explicit environment labels.

2. Context Learning

A context encoder processes the last $K$ transitions to produce a latent vector representing the current environment's properties. This vector is fed into all prediction heads, allowing the model to adapt its predictions based on recent experience.

3. Adaptive Planning

During inference, the agent must select the most appropriate specialized head. T-MCL employs an adaptive planning method that evaluates the prediction accuracy of each head over the most recent $N$ steps. The head with the lowest error is then used for Model Predictive Control (MPC) planning.

Overview of the T-MCL framework: (a) multi-headed dynamics model, (b) multiple choice learning, and (c) adaptive planning.

Experimental Results and Insights

T-MCL was evaluated on benchmark tasks including MuJoCo robotics (Hopper, HalfCheetah, CrippledAnt) and classic control. The agents were trained on specific parameter ranges and tested on unseen, out-of-distribution parameters to measure zero-shot generalization.

  • Superior Performance: T-MCL significantly outperformed state-of-the-art model-based methods like PETS and CaDM. Notably, on the CrippledAnt task, T-MCL achieved a 3.5x higher average return than CaDM.
  • Unsupervised Specialization: Visualizations confirm that the trajectory-wise loss effectively clusters physics. The model naturally assigns different heads to different physical configurations without explicit labels.
  • Efficiency: T-MCL matched or exceeded the performance of leading model-free meta-RL algorithms like PEARL while maintaining the sample efficiency characteristic of model-based approaches.

Conclusion

Trajectory-wise Multiple Choice Learning provides a powerful framework for dynamics generalization. By combining specialized multi-headed models with context-aware adaptation and adaptive planning, T-MCL successfully navigates the complexities of multi-modal environment dynamics. Future work may explore continuous blending of heads and handling non-stationary reward functions.

Summary updated

Abstract

Model-based reinforcement learning (RL) has shown great potential in various control tasks in terms of both sample-efficiency and final performance. However, learning a generalizable dynamics model robust to changes in dynamics remains a challenge since the target transition dynamics follow a multi-modal distribution. In this paper, we present a new model-based RL algorithm, coined trajectory-wise multiple choice learning, that learns a multi-headed dynamics model for dynamics generalization. The main idea is updating the most accurate prediction head to specialize each head in certain environments with similar dynamics, i.e., clustering environments. Moreover, we incorporate context learning, which encodes dynamics-specific information from past experiences into the context latent vector, enabling the model to perform online adaptation to unseen environments. Finally, to utilize the specialized prediction heads more effectively, we propose an adaptive planning method, which selects the most accurate prediction head over a recent experience. Our method exhibits superior zero-shot generalization performance across a variety of control tasks, compared to state-of-the-art RL methods. Source code and videos are available at https://sites.google.com/view/trajectory-mcl.
BibTeX
@inproceedings{seo2020trajectory,
  author = {Younggyo Seo and Kimin Lee and Ignasi Clavera and Thanard Kurutach and Jinwoo Shin and Pieter Abbeel},
  title = {{Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=BG4vr0SGfq},
}
Figure