HogRider: Champion Agent of Microsoft Malmo Collaborative AI Challenge

Yanhai Xiong, Haipeng Chen, Mengchen Zhao, Bo An

Abstract

HogRider is an autonomous agent designed for the 2017 Microsoft Malmo Collaborative AI Challenge. By integrating a noise-robust Bayesian framework for partner behavior prediction with a highly optimized Q-learning approach—utilizing human-reasoning "warm starts" and state abstraction—HogRider significantly outperformed both rival AI teams and human players in a complex Minecraft-based coordination game. This paper details the architectural innovations that allowed HogRider to achieve superior stability and performance in environments characterized by observation uncertainty and limited training trials.

Introduction

The challenge of ad hoc teamwork involves creating agents that can effectively collaborate with partners whose strategies are unknown and whose actions may be observed incorrectly. The Microsoft Malmo Collaborative AI Challenge (MCAC) utilized the "Pig Chase" mini-game to benchmark these capabilities. In this environment, two agents must decide whether to cooperate to trap a pig for a high reward or act selfishly to reach an exit for a lower, but guaranteed, reward.

The Pig Chase environment in Microsoft Malmo, showing both the first-person 3D view and the symbolic grid representation used by the agent.

The environment presents three primary hurdles: observation uncertainty (where the simulator occasionally misreports partner actions), limited learning trials due to slow simulation speeds, and a complex state space that makes standard reinforcement learning (RL) impractical.

Methodology

HogRider addresses these challenges through two primary components: a robust agent-type hypothesis framework and an optimized Q-learning policy.

Robust Agent Type Hypothesis

To collaborate effectively, HogRider must determine if its partner is "Focused" (actively chasing the pig) or "Random." Traditional Bayesian updates are fragile in the face of noisy observations. To mitigate this, the authors derived a generalized version of Bayes' theorem accounting for a ~25% observation error rate. Furthermore, they implemented a "Squash Bayes Update" using a hyperbolic tangent ($tanh$) function. This ensures that belief updates remain stable and prevents a single misobserved action from prematurely discarding a correct hypothesis.

Optimized Q-Learning

Standard Q-learning was adapted to handle the constraints of the Malmo environment through three key optimizations:

  1. State-Action Abstraction: By utilizing relative distances (e.g., the distance between the agent and the pig) rather than absolute coordinates, the authors reduced the state space from over 4 million possible configurations to just 60,000.
  2. Warm Start via Human Reasoning: Q-values were initialized using a decision tree based on human logic rather than random values. This "warm start" allowed the agent to exhibit competent behavior from the very first episode.
  3. Active-$\epsilon$-greedy Exploration: To avoid catastrophic moves during limited trials, HogRider only explores sub-optimal actions if their estimated value is at least 50% of the current best action's value.
Symbolic representations of the two primary strategies in the Pig Chase game: (a) catching a pig alone and (b) catching a pig together.

Evaluation and Results

HogRider was evaluated against 81 teams from 26 countries in the MCAC "Pig Chase" benchmark, as well as against a dataset of games played by 10 human PhD candidates.

  • Tournament Performance: HogRider secured first place, outperforming the runner-up by 13% in mean score. Notably, it demonstrated 21.7% better stability, a critical metric in a challenge where the highest average score determines the winner.
  • Human Comparison: HogRider outperformed human players by 28.1% in mean score, demonstrating the efficacy of combining human intuition with machine optimization.
  • Ablation Studies: The "Warm Start" initialization was found to improve the mean score by 10.9% over random initialization, while the Active-$\epsilon$-greedy strategy provided a 4.1% performance edge.

Conclusion

The success of HogRider highlights the importance of state abstraction and human-guided initialization in making reinforcement learning computationally feasible for complex tasks. By prioritizing stability through "Squash Bayes" updates and leveraging relative spatial relationships, HogRider proved that robust, ad hoc teamwork is achievable even under conditions of high uncertainty. Future work will explore extending these abstractions to 3D environments and scenarios with evolving partner strategies.

Summary updated

Abstract

It has been an open challenge for self-interested agents to make optimal sequential decisions in complex multiagent systems, where agents might achieve higher utility via collaboration. The Microsoft Malmo Collaborative AI Challenge (MCAC), which is designed to encourage research relating to various problems in Collaborative AI, takes the form of a Minecraft mini-game where players might work together to catch a pig or deviate from cooperation, for pursuing high scores to win the challenge. Various characteristics, such as complex interactions among agents, uncertainties, sequential decision making and limited learning trials all make it extremely challenging to find effective strategies. We present HogRider---the champion agent of MCAC in 2017 out of 81 teams from 26 countries. One key innovation of HogRider is a generalized agent type hypothesis framework to identify the behavior model of the other agents, which is demonstrated to be robust to observation uncertainty. On top of that, a second key innovation is a novel Q-learning approach to learn effective policies against each type of the collaborating agents. Various ideas are proposed to adapt traditional Q-learning to handle complexities in the challenge, including state-action abstraction to reduce problem scale, a warm start approach using human reasoning for addressing limited learning trials, and an active greedy strategy to balance exploitation-exploration. Challenge results show that HogRider outperforms all the other teams by a significant edge, in terms of both optimality and stability.
BibTeX
@inproceedings{xiong2018hogrider,
  author = {Yanhai Xiong and Haipeng Chen and Mengchen Zhao and Bo An},
  title = {{HogRider: Champion Agent of Microsoft Malmo Collaborative AI Challenge}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11581},
}
Figure