Introduction
In Reinforcement Learning (RL), agents typically optimize a numerical reward function. However, many real-world and complex virtual tasks lack explicit scoring mechanisms. The MineRL BASALT (Benchmark for Agents that Solve Almost-Lifelike Tasks) challenge highlights this difficulty by requiring agents to complete hierarchical, natural-language-defined tasks in Minecraft, such as "build a beautiful waterfall" or "create an animal pen." Traditional end-to-end learning methods often struggle with these tasks due to the requirements for long-term planning and the scarcity of expert data. This paper presents a "Hybrid Intelligence" approach that successfully addressed these challenges, winning the 2021 NeurIPS MineRL BASALT competition.
The Hybrid Intelligence Architecture
The core innovation is the integration of human knowledge engineering with machine learning. Rather than a monolithic neural network, the agent's architecture is modular, combining specialized components for perception, navigation, and high-level logic.
1. State Classification (The Eyes)
To understand its surroundings, the agent uses a Convolutional Neural Network (CNN) trained as a state classifier. This module identifies 13 distinct environmental states relevant to the tasks. The classifier was trained on a custom dataset of over 80,000 human-labeled image frames, allowing the agent to recognize specific contexts like being "inside a cave" or having a "good waterfall view."
2. Navigation via Behavior Cloning (The Legs)
For movement, the researchers employed Behavior Cloning (BC). By training on human demonstration data provided in the MineRL-v0 dataset, the agent learned to navigate Minecraft's varied terrain in a manner that mimics human movement patterns.
3. Estimated Odometry (The Compass)
Because the competition restricted access to ground-truth coordinates, the team implemented an engineered odometry module. By estimating displacement based on known walking speeds and camera orientations, the agent could maintain a rough internal map. This allowed it to "remember" locations, such as where it previously spotted animals, enabling it to return to them—a capability often lacking in pure reactive agents.
4. Hierarchical State-Machine (The Logic)
The high-level decision-making is governed by a hand-coded state machine. This component acts as the "brain," using inputs from the state classifier and odometry to transition between subtasks. For instance, if the goal is to build a waterfall, the state machine might trigger a "climb mountain" subtask upon detecting a mountain, followed by a "build" sequence once the agent reaches the summit.
Results and Performance
The hybrid approach proved significantly more effective than pure learning-based methods. In pairwise human evaluations, the Hybrid agent achieved a TrueSkill rating of 25.49, outperforming both pure Behavior Cloning (20.30) and a baseline engineered random-walk agent (24.21).
Notably, in the "MakeWaterfall" task, the Hybrid agent occasionally outperformed human players in terms of efficiency. While humans often spent considerable time searching for an aesthetically "perfect" location, the agent's structured logic allowed it to identify a suitable spot and complete the task more decisively. Furthermore, the hybrid agent was frequently rated as the "most human-like," suggesting that structured, goal-oriented behavior appears more natural to observers than the jittery movements typical of pure imitation learning.
Discussion and Limitations
While successful, the system has limitations. The estimated odometry is subject to "drift" over time, especially when the agent encounters obstacles that impede movement without changing visual input. Additionally, the agent's reliance on a low-resolution (64x64) input limits the precision of the state classifier.
The primary drawback of this approach is the manual effort required to design the state machine for each task. Future research aims to develop "Informed AI" that can automatically derive these hierarchical structures from natural language instructions, reducing the need for manual knowledge engineering while retaining the benefits of a structured architecture.
Summary updated