Combining Learning from Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks in Minecraft

Vinicius G. Goecks, Nicholas R. Waytowich, David Watkins, Bharat Prakash

Introduction

In Reinforcement Learning (RL), agents typically optimize a numerical reward function. However, many real-world and complex virtual tasks lack explicit scoring mechanisms. The MineRL BASALT (Benchmark for Agents that Solve Almost-Lifelike Tasks) challenge highlights this difficulty by requiring agents to complete hierarchical, natural-language-defined tasks in Minecraft, such as "build a beautiful waterfall" or "create an animal pen." Traditional end-to-end learning methods often struggle with these tasks due to the requirements for long-term planning and the scarcity of expert data. This paper presents a "Hybrid Intelligence" approach that successfully addressed these challenges, winning the 2021 NeurIPS MineRL BASALT competition.

The Hybrid Intelligence Architecture

The core innovation is the integration of human knowledge engineering with machine learning. Rather than a monolithic neural network, the agent's architecture is modular, combining specialized components for perception, navigation, and high-level logic.

Overview of the Hybrid Intelligence architecture, showing the flow from environmental input to action selection via state machines and learned policies.

1. State Classification (The Eyes)

To understand its surroundings, the agent uses a Convolutional Neural Network (CNN) trained as a state classifier. This module identifies 13 distinct environmental states relevant to the tasks. The classifier was trained on a custom dataset of over 80,000 human-labeled image frames, allowing the agent to recognize specific contexts like being "inside a cave" or having a "good waterfall view."

Examples of the 13 environmental states used to train the state classifier, ranging from terrain features to task-specific views.

2. Navigation via Behavior Cloning (The Legs)

For movement, the researchers employed Behavior Cloning (BC). By training on human demonstration data provided in the MineRL-v0 dataset, the agent learned to navigate Minecraft's varied terrain in a manner that mimics human movement patterns.

3. Estimated Odometry (The Compass)

Because the competition restricted access to ground-truth coordinates, the team implemented an engineered odometry module. By estimating displacement based on known walking speeds and camera orientations, the agent could maintain a rough internal map. This allowed it to "remember" locations, such as where it previously spotted animals, enabling it to return to them—a capability often lacking in pure reactive agents.

4. Hierarchical State-Machine (The Logic)

The high-level decision-making is governed by a hand-coded state machine. This component acts as the "brain," using inputs from the state classifier and odometry to transition between subtasks. For instance, if the goal is to build a waterfall, the state machine might trigger a "climb mountain" subtask upon detecting a mountain, followed by a "build" sequence once the agent reaches the summit.

Results and Performance

The hybrid approach proved significantly more effective than pure learning-based methods. In pairwise human evaluations, the Hybrid agent achieved a TrueSkill rating of 25.49, outperforming both pure Behavior Cloning (20.30) and a baseline engineered random-walk agent (24.21).

Notably, in the "MakeWaterfall" task, the Hybrid agent occasionally outperformed human players in terms of efficiency. While humans often spent considerable time searching for an aesthetically "perfect" location, the agent's structured logic allowed it to identify a suitable spot and complete the task more decisively. Furthermore, the hybrid agent was frequently rated as the "most human-like," suggesting that structured, goal-oriented behavior appears more natural to observers than the jittery movements typical of pure imitation learning.

Discussion and Limitations

While successful, the system has limitations. The estimated odometry is subject to "drift" over time, especially when the agent encounters obstacles that impede movement without changing visual input. Additionally, the agent's reliance on a low-resolution (64x64) input limits the precision of the state classifier.

The primary drawback of this approach is the manual effort required to design the state machine for each task. Future research aims to develop "Informed AI" that can automatically derive these hierarchical structures from natural language instructions, reducing the need for manual knowledge engineering while retaining the benefits of a structured architecture.

Summary updated

BibTeX
@inproceedings{goecks2022combining,
  author = {Vinicius G. Goecks and Nicholas R. Waytowich and David Watkins and Bharat Prakash},
  title = {{Combining Learning from Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks in Minecraft}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
}
Figure