Abstract
Learning rational behaviors in open-world games like Minecraft remains challenging for Reinforcement Learning (RL) due to partial observability, high-dimensional visual perception, and delayed rewards. We propose JueWu-MC, a sample-efficient hierarchical RL approach equipped with representation learning and imitation learning. Our approach includes a two-level hierarchy: a high-level meta-policy for sub-task selection and low-level workers for sub-task execution. We introduce action-aware representation learning, discriminator-based self-imitation learning, and ensemble behavior cloning with consistency filtering. JueWu-MC significantly improves sample efficiency, winning the NeurIPS MineRL 2021 competition with the highest performance score ever recorded.
Introduction
Training RL agents for complex 3D environments like Minecraft involves overcoming several hurdles. The long-time horizons and sparse rewards associated with tasks like mining diamonds require chaining numerous dependent sub-tasks. High-dimensional visual perception from a first-person view adds complexity, while standard exploration strategies often fail to find meaningful signals in the vast state space. Furthermore, while human demonstrations are available, they are often noisy and unstructured.
The MineRL competition environment exacerbates these issues by limiting training to 8 million samples and obfuscating state and action spaces. JueWu-MC addresses these challenges by extracting useful priors from human data through a highly sample-efficient hierarchical framework.
Methodology
JueWu-MC utilizes a two-level Hierarchical RL (HRL) framework. A high-level meta-policy extracts sub-goals from unstructured human demonstrations and manages the execution of specific options. Low-level sub-policies are then trained to execute individual sub-tasks.
To enhance sub-policy training, three core techniques are employed:
- Action-aware Representation Learning (A2RL): A self-supervised method that learns a mask network to focus on environment changes caused by specific actions, creating compact and interpretable representations.
- Discriminator-based Self-imitation Learning (DSIL): This replaces standard advantage-based methods by using a discriminator to distinguish between successful and failed trajectories, providing intrinsic rewards that guide the agent toward productive state distributions.
- Ensemble Behavior Cloning with Consistency Filtering (EBC): For crafting tasks, noisy human data is filtered for consistency. An ensemble of policies is then trained on subsets of this data, using majority voting during inference to reduce compounding errors.
Experimental Results
JueWu-MC demonstrated superior performance in the NeurIPS MineRL 2021 competition, achieving a score of 76.97—over 3.4 times higher than the runner-up. The agent reached a score of 100 with only 2.5 million samples, showcasing extreme sample efficiency.
Ablation studies confirm the necessity of each component. While A2RL and EBC provide significant early-stage boosts, DSIL is critical for late-stage tasks involving hard exploration, such as locating iron ore. The framework's ability to stay within "good" state distributions through intrinsic rewards makes the policy self-correcting and robust.
Conclusion
JueWu-MC effectively masters the complexities of Minecraft by combining hierarchical control with advanced representation and imitation learning. By filtering noisy human data and focusing on action-relevant visual features, the agent achieves unprecedented sample efficiency. Future work will focus on extending this framework to a wider variety of Minecraft tasks and other complex open-world games.
Summary updated