JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learning

Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, Wei Yang

Abstract

Learning rational behaviors in open-world games like Minecraft remains challenging for Reinforcement Learning (RL) due to partial observability, high-dimensional visual perception, and delayed rewards. We propose JueWu-MC, a sample-efficient hierarchical RL approach equipped with representation learning and imitation learning. Our approach includes a two-level hierarchy: a high-level meta-policy for sub-task selection and low-level workers for sub-task execution. We introduce action-aware representation learning, discriminator-based self-imitation learning, and ensemble behavior cloning with consistency filtering. JueWu-MC significantly improves sample efficiency, winning the NeurIPS MineRL 2021 competition with the highest performance score ever recorded.

Introduction

Training RL agents for complex 3D environments like Minecraft involves overcoming several hurdles. The long-time horizons and sparse rewards associated with tasks like mining diamonds require chaining numerous dependent sub-tasks. High-dimensional visual perception from a first-person view adds complexity, while standard exploration strategies often fail to find meaningful signals in the vast state space. Furthermore, while human demonstrations are available, they are often noisy and unstructured.

The MineRL competition environment exacerbates these issues by limiting training to 8 million samples and obfuscating state and action spaces. JueWu-MC addresses these challenges by extracting useful priors from human data through a highly sample-efficient hierarchical framework.

Methodology

JueWu-MC utilizes a two-level Hierarchical RL (HRL) framework. A high-level meta-policy extracts sub-goals from unstructured human demonstrations and manages the execution of specific options. Low-level sub-policies are then trained to execute individual sub-tasks.

The JueWu-MC hierarchical reinforcement learning framework.

To enhance sub-policy training, three core techniques are employed:

  1. Action-aware Representation Learning (A2RL): A self-supervised method that learns a mask network to focus on environment changes caused by specific actions, creating compact and interpretable representations.
  2. Discriminator-based Self-imitation Learning (DSIL): This replaces standard advantage-based methods by using a discriminator to distinguish between successful and failed trajectories, providing intrinsic rewards that guide the agent toward productive state distributions.
  3. Ensemble Behavior Cloning with Consistency Filtering (EBC): For crafting tasks, noisy human data is filtered for consistency. An ensemble of policies is then trained on subsets of this data, using majority voting during inference to reduce compounding errors.

Experimental Results

JueWu-MC demonstrated superior performance in the NeurIPS MineRL 2021 competition, achieving a score of 76.97—over 3.4 times higher than the runner-up. The agent reached a score of 100 with only 2.5 million samples, showcasing extreme sample efficiency.

Performance comparison and ablation study of JueWu-MC.

Ablation studies confirm the necessity of each component. While A2RL and EBC provide significant early-stage boosts, DSIL is critical for late-stage tasks involving hard exploration, such as locating iron ore. The framework's ability to stay within "good" state distributions through intrinsic rewards makes the policy self-correcting and robust.

Conclusion

JueWu-MC effectively masters the complexities of Minecraft by combining hierarchical control with advanced representation and imitation learning. By filtering noisy human data and focusing on action-relevant visual features, the agent achieves unprecedented sample efficiency. Future work will focus on extending this framework to a wider variety of Minecraft tasks and other complex open-world games.

Summary updated

Abstract

Learning rational behaviors in open-world games like Minecraft remains to be challenging for Reinforcement Learning (RL) research due to the compound challenge of partial observability, high-dimensional visual perception and delayed reward. To address this, we propose JueWu-MC, a sample-efficient hierarchical RL approach equipped with representation learning and imitation learning to deal with perception and exploration. Specifically, our approach includes two levels of hierarchy, where the high-level controller learns a policy to control over options and the low-level workers learn to solve each sub-task. To boost the learning of sub-tasks, we propose a combination of techniques including 1) action-aware representation learning which captures underlying relations between action and representation, 2) discriminator-based self-imitation learning for efficient exploration, and 3) ensemble behavior cloning with consistency filtering for policy robustness. Extensive experiments show that JueWu-MC significantly improves sample efficiency and outperforms a set of baselines by a large margin. Notably, we won the championship of the NeurIPS MineRL 2021 research competition and achieved the highest performance score ever.
BibTeX
@inproceedings{lin2022juewu,
  author = {Zichuan Lin and Junyou Li and Jianing Shi and Deheng Ye and Qiang Fu and Wei Yang},
  title = {{JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=FcqZtkoH44},
}
Figure