Reward-Free Curricula for Training Robust World Models

Marc Rigter, Minqi Jiang, Ingmar Posner

Abstract

WAKER (Weighted Acquisition of Knowledge across Environments for Robustness) is a novel framework for reward-free automatic curriculum learning. By theoretically linking model prediction error to minimax regret, WAKER generates a curriculum that focuses training on environments where the world model is most uncertain. This approach leads to agents that are significantly more robust and generalize better to out-of-distribution scenarios without requiring any task-specific reward signal. Experimental results across continuous control domains demonstrate that WAKER significantly enhances agent robustness and out-of-distribution generalization compared to standard domain randomization.

Introduction

The quest for generally capable agents requires models that can adapt to new tasks in diverse environments without constant retraining. A popular approach is reward-free pre-training, where an agent explores an environment to learn a "world model"—a predictive simulator of physics and dynamics. Once learned, the agent can "imagine" thousands of new tasks within this model to find optimal policies. However, a world model trained in one specific setting often fails when environment dynamics change. While Domain Randomization (DR) is a common fix, it is often inefficient. Unsupervised Environment Design (UED) offers a better way by creating a curriculum of increasingly difficult environments, but existing UED methods require a task-specific reward signal. WAKER solves this "chicken-and-egg" problem by building a curriculum for a robust world model without a reward function.

Methodology

The WAKER framework is built upon a solid theoretical foundation that bridges the gap between intrinsic motivation and environment design.

WAKER framework overview and latent space representation.

Theoretical Foundation

The authors prove that minimizing the maximum expected error of the world model across all possible environment instances provides an upper bound on the minimax regret. In essence, if a model is accurate even in its "worst" environment, the policies trained inside it will be robust.

Error Estimation and Curriculum

Since the true error is unknown during training, WAKER utilizes an ensemble of latent dynamics models. The disagreement (variance) between these models serves as a proxy for epistemic uncertainty. WAKER maintains a buffer of these error estimates for different environment parameters (e.g., terrain roughness, arena size) and uses a Boltzmann distribution to sample the next environment for data collection. This biases the agent toward environments where the world model currently performs the worst. Two variants were explored: WAKER-M, which prioritizes environments with the highest magnitude of estimated error, and WAKER-R, which prioritizes environments with the highest rate of reduction in error.

Experimental Evaluation

WAKER was evaluated on several pixel-based continuous control domains, including Terrain Walker, Terrain Hopper, and Clean Up tasks. To test generalization, the authors introduced out-of-distribution (OOD) benchmarks such as "Steep" terrain and "Extra Block" scenarios.

The Terrain Walker environment used for evaluating robustness.

Key Results and Discussion

  • Superior Robustness: WAKER-M consistently outperformed Domain Randomization and other baselines. In the Terrain Walker task, WAKER achieved significantly higher CVaR₀.₁, indicating much higher reliability in the most challenging environments.
  • Stronger Generalization: WAKER showed a massive lead in OOD tests. For example, in the Clean Up Extra Block task, WAKER-M achieved a task completion score of ~0.53 compared to ~0.33 for standard Domain Randomization.
  • Efficiency: WAKER reaches higher performance levels with fewer environment steps because it avoids wasting time on simple environments the model already understands.
  • Robustness at No Cost to Average Performance: Unlike some robust methods that sacrifice peak performance to handle edge cases, WAKER often improves average performance because the curriculum forces the model to master complex dynamics that uniform sampling might ignore.

Conclusion

WAKER demonstrates that reward-free curriculum learning is a powerful tool for building robust world models. By using model uncertainty as a proxy for regret, it automates the discovery of difficult environment regions. Future work will focus on scaling WAKER to high-dimensional parameter spaces and reducing the computational cost associated with training ensembles of world models.

Summary updated

BibTeX
@inproceedings{rigter2024reward,
  author = {Marc Rigter and Minqi Jiang and Ingmar Posner},
  title = {{Reward-Free Curricula for Training Robust World Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=eCGpNGDeNu},
}
Figure