Abstract: MineRL 2019 competition challenged participants to train sample-efficient agents to play Minecraft, by using a dataset of human gameplay and a limit number of steps the environment. We approached this task with behavioural cloning by predicting what actions human players would take, and reached fifth place in the final ranking. Despite being a simple algorithm, we observed the performance of such an approach can vary significantly, based on when the training is stopped. In this paper, we detail our submission to the competition, run further experiments to study how performance varied over training and study how different engineering decisions affected these results.
Introduction
Reinforcement Learning (RL) is notoriously sample-inefficient and highly sensitive to hyperparameter tuning, making it difficult to train robust agents for complex, open-ended games like Minecraft. The MineRL 2019 competition challenged researchers to train an agent to obtain a diamond in Minecraft using a limited environment training budget and a large dataset of human gameplay.
To avoid the instability of RL, we opted for a pure imitation learning approach: Behavioural Cloning (BC). However, we discovered a gap in the existing literature. BC results are often reported as single, averaged numbers, painting it as a highly stable supervised learning method. In reality, BC suffers from significant variance, causal confusion, and class imbalance. This paper documents the engineering required to make a pure BC approach competitive.
Methods
We treated playing Minecraft as a multilabel classification problem, predicting the discrete actions a human would take given a specific visual observation and inventory state.
- Architecture: We used a deep neural network based on the IMPALA architecture. It processes 64x64 RGB images through three Residual blocks, concatenates the output with "direct features" (inventory counts and equipped items), and passes them through fully connected layers to output action probabilities.
- No Recurrence: Notably, we excluded recurrent layers (like LSTMs) and past frames. We found that providing memory actually degraded performance, likely due to causal confusion (e.g., the agent getting confused by dropped inputs or pausing).
- Replay Buffer for BC: Because the training data consists of sequential video frames, standard batch sampling leads to highly correlated data. To fix this, we asynchronously loaded samples into a massive 500,000-sample replay buffer and sampled training batches from there to stabilize learning.
- Label Smoothing & Augmentation: To combat overfitting and class imbalance, we applied label smoothing (0.005) and heavily tuned data augmentations (brightness, contrast, noise, horizontal flipping).
Datasets
The model was trained on the MineRL dataset, a large-scale collection of human Minecraft demonstrations. Specifically, we utilized the MineRLObtainDiamond-v0 and MineRLObtainIronPickaxe-v0 subsets, which include video frames, inventory states, and corresponding human keystrokes/mouse movements.
Key Insights and Ideas
BC is Not as Stable as You Think
We found massive run-to-run variance in our BC agents. Depending on the exact moment training was stopped, average reward could jump from 9.5 to 13.1 (a 38% relative increase). This suggests that BC research should report variance over multiple runs, similar to standard RL practices.
The "Crafting" Bottleneck
In Minecraft, players spend the vast majority of time moving and mining, but must occasionally craft items to progress. Crucial actions like "craft a wooden pickaxe" or "place a crafting table" appeared in less than 0.1% of the dataset. The agent easily learned to gather wood but often failed to place a crafting table simply because the action was so rare in the training data.
Data Augmentation Sensitivity
While standard computer vision augmentations are common, applying them haphazardly destroyed performance. We had to reduce the augmentation strengths until the changes were barely visible to the human eye to see consistent improvements.
Key Results
Our pure BC approach secured 5th place out of 40 participants in the final round of the MineRL 2019 competition. The Round 1 submission achieved an average reward of 21.7, while the Round 2 submission scored 17.9 (averaged over 100 evaluation games).
To prove that rare actions were the limiting factor, we ran an experiment where we hard-coded crafting actions but left everything else to the BC agent. This assist boosted evaluation results by 30% to 92%, with one agent reaching a reward of 291 in several games.
Limitations and Future Work
The primary limitation is the model's inability to reliably execute under-represented actions due to class imbalance. Weighting the loss function or under-sampling common actions did not improve results in our tests. Future work should explore more sophisticated data sampling techniques, such as SMOTE, to artificially boost the representation of rare bottleneck actions.
Summary updated