JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learning

Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, Wei Yang

Abstract: Learning rational behaviors in open-world games like Minecraft remains to be challenging for Reinforcement Learning (RL) research due to the compound challenge of partial observability, high-dimensional visual perception and delayed reward. To address this, we propose JueWu-MC, a sample-efficient hierarchical RL approach equipped with representation learning and imitation learning to deal with perception and exploration. Specifically, our approach includes two levels of hierarchy, where the high-level controller learns a policy to control over options and the low-level workers learn to solve each sub-task. To boost the learning of sub-tasks, we propose a combination of techniques including 1) action-aware representation learning which captures underlying relations between action and representation, 2) discriminator-based self-imitation learning for efficient exploration, and 3) ensemble behavior cloning with consistency filtering for policy robustness. Extensive experiments show that JueWu-MC significantly improves sample efficiency and outperforms a set of baselines by a large margin. Notably, we won the championship of the NeurIPS MineRL 2021 research competition and achieved the highest performance score ever.

Introduction

Reinforcement Learning (RL) agents typically struggle in open-world environments like Minecraft due to a "compound challenge": long-time horizons with sparse rewards, high-dimensional raw pixel perception, and inefficient exploration in massive 3D worlds. Furthermore, while human demonstrations are available, they are often noisy and inconsistent, complicating standard imitation learning approaches. JueWu-MC addresses these issues through a hierarchical framework that combines action-aware representation learning, discriminator-based self-imitation, and robust behavior cloning.

Methodology

JueWu-MC utilizes a two-level hierarchy consisting of a high-level meta-policy that selects sub-goals (e.g., "log" or "craft") and low-level workers that execute specific actions to reach those goals. To enhance the performance of these workers, three core innovations were introduced:

  • Action-aware Representation Learning (A2RL): This self-supervised approach uses a mask generator to identify which parts of the visual field are affected by specific actions. By focusing on these salient features, the agent can ignore visual noise and better understand how its actions change the environment.
  • Discriminator-based Self-imitation Learning (DSIL): Unlike standard self-imitation, DSIL trains a discriminator to distinguish between successful and failed trajectories. It provides intrinsic rewards for staying within the state distribution of successful runs, pushing the agent to self-correct and recover from mistakes.
  • Ensemble Behavior Cloning (EBC) with Consistency Filtering: To handle imperfect human data, the system filters demonstrations for consistent action sequences. An ensemble of policies then uses majority voting during inference to reduce uncertainty and prevent compounding errors.
Sub-policy architecture featuring A2RL and DSIL modules.

Experimental Setup

The framework was evaluated on the "ObtainDiamond" task within the MineRL environment. This task requires a complex, multi-stage sequence of resource gathering and crafting, making it a benchmark for long-horizon planning. Training utilized the MineRL dataset of human demonstrations.

Results and Discussion

JueWu-MC won the NeurIPS MineRL 2021 competition with a score of 76.97, outperforming the runner-up by a factor of 3.4.

Performance comparison against baselines and ablation study results.

Key results include:

  • Sample Efficiency: The agent reached a score of 100 within just 2.5 million training samples, whereas baseline algorithms like Rainbow or SQIL failed to achieve significant scores within the competition's 8-million-sample limit.
  • Component Efficacy: Ablation studies (Figure 6b) confirm that A2RL, DSIL, and EBC are all critical for maintaining performance. Without these components, the agent's ability to progress through the game's stages drops significantly.
  • Robustness: JueWu-MC maintained high success rates (87% to 92%) through the first four stages of the diamond-gathering pipeline, demonstrating superior stability compared to previous state-of-the-art models.

Conclusion and Future Work

JueWu-MC demonstrates that hierarchical RL, when combined with action-centric vision and robust self-imitation, can master complex open-world tasks with high sample efficiency. Future work will explore the generalization of A2RL and DSIL to other open-world domains and investigate the potential of fine-tuning the high-level meta-policy with reinforcement learning to further improve performance.

Summary updated

Abstract

Learning rational behaviors in open-world games like Minecraft remains to be challenging for Reinforcement Learning (RL) research due to the compound challenge of partial observability, high-dimensional visual perception and delayed reward. To address this, we propose JueWu-MC, a sample-efficient hierarchical RL approach equipped with representation learning and imitation learning to deal with perception and exploration. Specifically, our approach includes two levels of hierarchy, where the high-level controller learns a policy to control over options and the low-level workers learn to solve each sub-task. To boost the learning of sub-tasks, we propose a combination of techniques including 1) action-aware representation learning which captures underlying relations between action and representation, 2) discriminator-based self-imitation learning for efficient exploration, and 3) ensemble behavior cloning with consistency filtering for policy robustness. Extensive experiments show that JueWu-MC significantly improves sample efficiency and outperforms a set of baselines by a large margin. Notably, we won the championship of the NeurIPS MineRL 2021 research competition and achieved the highest performance score ever.
BibTeX
@inproceedings{lin2022juewu,
  author = {Zichuan Lin and Junyou Li and Jianing Shi and Deheng Ye and Qiang Fu and Wei Yang},
  title = {{JueWu-MC: Playing Minecraft with Sample-efficient Hierarchical Reinforcement Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=FcqZtkoH44},
}
Figure