Combining Learning From Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks in Minecraft

Vinicius G. Goecks, Nicholas R. Waytowich, David Watkins-Valls, Bharat Prakash

Abstract

The 2021 NeurIPS MineRL BASALT competition challenged participants to develop AI agents capable of solving complex, "human-like" tasks in Minecraft—such as building a waterfall or finding a cave—without explicit reward functions. Researchers from the Army Research Laboratory and Columbia University secured first place by employing a "Hybrid Intelligence" approach. This architecture integrates behavior cloning and visual state classification with classical state-machine logic and engineered odometry. Their results demonstrate that combining learned perception with structured engineering effectively addresses the long-term planning and common-sense reasoning required for open-ended hierarchical tasks.

Introduction

Most contemporary AI breakthroughs in gaming, such as those in Go or StarCraft II, rely on reinforcement learning (RL) where an agent maximizes a numerical score. However, real-world tasks often lack clear mathematical reward signals. The MineRL BASALT (Benchmark for Agents that Solve Almost-Lifelike Tasks) competition addresses this by requiring agents to complete four tasks defined only by natural language: finding a cave, building a waterfall, creating an animal pen, and building a house.

Pure end-to-end machine learning often struggles with these tasks due to the high-dimensional state space (raw pixels) and the need for long-term planning. To overcome these hurdles, the authors proposed a hybrid system that balances the flexibility of learned models with the reliability of engineered logic.

Methods: The Hybrid Intelligence Architecture

The core of the winning approach is a multi-component system that splits the workload between perception, movement, and high-level decision-making.

Overview of the Hybrid Intelligence architecture.

State Classification (Learned)

To provide the agent with environmental awareness, the team developed a custom image classifier. They manually labeled 81,888 image frames across 13 distinct classes representing critical environmental states. A convolutional neural network was trained to detect these states in real-time, allowing the agent to "see" relevant features such as caves, animals, or open spaces.

Examples of the 13 environmental states identified by the custom classifier.

Human-like movement was achieved through Behavior Cloning (BC). By training on human demonstration data provided by the competition, the agent learned to traverse Minecraft’s complex terrain in a natural manner, avoiding the erratic movements often seen in pure RL agents.

Estimated Odometry (Engineered)

Because the competition rules prohibited the use of ground-truth coordinates, the researchers implemented a "dead reckoning" system. By tracking the simulator's frame rate and standard movement speeds, the agent estimates its position and heading. This allows it to maintain a rudimentary mental map of the environment, noting locations of interest like open spaces or animal sightings.

State-Machine (Engineered)

The high-level "brain" of the agent is a state machine. It consumes the outputs from the visual classifier and the odometry system to manage task hierarchies. For instance, in the animal pen task, the state machine coordinates transitions between "Search for flat ground," "Build pen," and "Lure animals" based on visual confirmation of these states.

Results and Key Insights

The agents were evaluated by human judges using the TrueSkill rating system, which measures performance and "human-likeness."

  • Performance: The Hybrid agent won the official competition, consistently outperforming both pure Behavior Cloning and pure Engineering baselines. In the FindCave task, the Hybrid agent achieved a TrueSkill of ~25.14, compared to ~24.32 for BC and ~24.29 for Engineering.
  • Efficiency vs. Aesthetics: Interestingly, the agent occasionally outperformed humans in speed. In the MakeWaterfall task, while humans spent time searching for aesthetically pleasing locations, the agent’s logic was satisfied by any valid mountain peak.
  • Safety Engineering: The inclusion of hard-coded "safety" routines, such as an "escape water" subtask, significantly improved the agent's perceived human-likeness by preventing it from getting stuck in common environmental traps.

Limitations and Future Work

A primary technical challenge remains "odometry drift." Since the agent estimates its position based on actions rather than ground truth, inaccuracies accumulate over time—especially if the agent is obstructed by a wall. Future improvements could involve integrating visual SLAM (Simultaneous Localization and Mapping) or more robust "stuck detection" mechanisms. Additionally, while the current state machine is hand-coded, future research may explore learning these hierarchies directly from natural language instructions.

Summary updated

BibTeX
@inproceedings{goecks2022combining,
  author = {Vinicius G. Goecks and Nicholas R. Waytowich and David Watkins-Valls and Bharat Prakash},
  title = {{Combining Learning From Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks in Minecraft}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
}
Figure