Introduction
Reinforcement Learning (RL) is notoriously sample-inefficient, often requiring millions of interactions with an environment to learn basic tasks. The MineRL 2019 competition challenged researchers to train agents to find a diamond in Minecraft—a task involving a complex hierarchy of resource gathering and crafting—using a limited training budget and a dataset of human gameplay.
While many participants turned to complex RL algorithms, this paper investigates whether Behavioural Cloning (BC), a form of imitation learning where a model is trained to mimic human actions, can handle the challenge. The authors specifically tackle the "robustness myth" of BC, exploring why a seemingly simple supervised learning task often produces inconsistent results in practice. Ultimately, they demonstrate that simple BC can be surprisingly competitive in complex, hierarchical environments, securing 5th place in the competition. However, they reveal that BC is far from a "plug-and-play" solution, requiring meticulous engineering to manage high performance variance and dataset imbalances.
Methods
The authors treated the problem as a multi-label classification task, utilizing the MineRL 2019 dataset (specifically the MineRLObtainDiamond-v0 and MineRLObtainIronPickaxe-v0 subsets), which consists of thousands of recorded human play sessions.
Their approach included several key design choices:
- Model Architecture: They employed a deep residual network (ResNet) based on the IMPALA architecture. It processes 64x64 RGB images alongside "direct features" like inventory counts and item durability.
- Action Space Engineering: The Minecraft action space was simplified by removing redundant movements (like sneaking or moving backward) and discretizing continuous mouse movements into fixed-degree turns.
- No Memory: Interestingly, the authors opted not to use recurrent layers (LSTMs) or frame stacking. They found that providing past information actually hindered performance, likely due to "causal confusion" where the model misattributes current actions to past events rather than current visual cues.
- Training Stabilizers: To combat the correlated nature of video data (where consecutive frames are nearly identical), they used a large replay buffer of 500,000 samples to ensure training batches were diverse and independent.
Key Insights and Ideas
The "Stopping Point" Paradox
Unlike standard supervised learning where performance usually plateaus smoothly, the authors found that BC performance in Minecraft fluctuates wildly.
Stopping training at a "lucky" spike could result in a 38% higher reward than stopping just a few iterations later. During training, the mean episodic reward was observed to jump from 9.5 to 13.1 simply by selecting the optimal checkpoint, highlighting the extreme variance inherent in the process.
Causal Confusion and Data Sampling
The decision to omit LSTMs is a major takeaway. In BC, models often over-rely on past actions to predict the next one, failing to learn the actual visual cues required to navigate the environment. Furthermore, standard data loaders read consecutive frames, leading to highly correlated batches that cause oscillations in the training loss. A large asynchronous replay buffer proved essential to "de-correlate" the human demonstrations.
The "Rare Action" Bottleneck
In Minecraft, a player might spend 5,000 frames mining wood but only 1 frame placing a crafting table. Even with label smoothing, the model struggles to prioritize these rare but critical "bottleneck" actions.
Results and Limitations
The system reached 5th place out of 40 participants in the final round of the MineRL competition. The agent successfully learned to gather wood and craft planks and sticks reliably. However, it struggled with the physical act of placing the crafting table. When the authors supplemented the BC agent with hard-coded crafting logic in an ablation study, rewards increased by 30% to 92%, proving that the model's primary failure was the "last mile" execution of rare actions.
The primary limitation of this approach is distributional shift: once the agent makes a small mistake, it enters a state (e.g., staring at a wall) that it never saw in the human training data, leading to a total failure.
The authors suggest that future work should focus on more sophisticated sampling techniques to handle class imbalance and exploring ways to report and manage the high variance inherent in imitation learning. They conclude that while BC is a powerful baseline, it is not yet a robust "set-and-forget" method for complex 3D environments.
Summary updated