The Arcade Learning Environment: An Evaluation Platform for General Agents (Extended Abstract)

Marc G. Bellemare, Yavar Naddaf, Joel Veness, Michael BowlingIJCAI 2015 · OpenReview

Introduction

A fundamental goal of artificial intelligence is the creation of "generally competent" agents—algorithms capable of performing well across a wide array of tasks without requiring domain-specific tuning. Historically, AI research has often relied on narrow benchmarks, leading to "method overfitting," where algorithms are inadvertently optimized for specific problem quirks. To address this, we introduce the Arcade Learning Environment (ALE), a platform providing an interface to hundreds of Atari 2600 games. These games are diverse, designed for human players, and offer a rigorous testbed for evaluating general AI technology.

The ALE Framework and Methodology

The ALE interfaces with the Stella emulator, treating each Atari game as a reinforcement learning (RL) problem. Agents receive screen pixels or RAM state as input and must select from up to 18 joystick actions to maximize the game score. This setup challenges agents to learn from high-dimensional visual input and sparse rewards.

We evaluated two primary paradigms:

  1. Reinforcement Learning: Using SARSA($\lambda$) with various feature-generation methods, including binary color encodings (Basic/BASS), Locally Sensitive Hashing (LSH), and object detection (DISCO).
  2. Planning: Using Monte Carlo Tree Search (specifically UCT), where the agent uses the emulator as a generative model to simulate future states.

To ensure a fair evaluation of generality, we tuned hyperparameters on a small set of 5 games and then tested the agents on a semi-randomly selected suite of 50 additional games.

Empirical Results

Our evaluation across 55 games revealed significant insights into the current state of general AI. Learning agents outperformed baseline random agents on 40 of the 55 games. Among the RL methods, the BASS representation—which utilizes pairwise feature combinations—demonstrated the most robust median performance across the suite.

Average and median baseline scores across 55 games for various RL feature representations.

Planning agents, particularly UCT, showed even greater success, outperforming both learning and baseline agents on 49 out of 55 games. However, this performance comes at a high computational cost, requiring significant look-ahead time per action. As shown in the table below, UCT often achieves scores orders of magnitude higher than the best learning agents.

Comparison of planning (UCT) versus learning (Best Learner) performance on selected Atari games.

Challenges and The "Determinism Trap"

A key finding was the success of "The Brute," a simple trajectory optimization algorithm that outperformed learning methods on 45 games. This highlighted a critical issue: the deterministic nature of the Atari emulator allowed agents to "win" through memorization rather than true learning. To move toward general intelligence, future benchmarks must incorporate stochasticity—such as "sticky actions"—to prevent agents from relying on fixed action sequences.

Conclusion

The Arcade Learning Environment provides a necessary shift toward evaluating general competency in AI. While current agents show promise, the gap between AI and human-level performance remains significant, particularly in games requiring long-term planning and handling sparse rewards. ALE serves as a foundational platform for the next generation of reinforcement learning and planning research.

Summary updated

Abstract

In this extended abstract we introduce the Arcade Learning Environment (ALE): both a challenge problem and a platform and methodology for evaluating the development of general, domain-independent AI technology. ALE provides an interface to hundreds of Atari 2600 game environments, each one different, interesting, and designed to be a challenge for human players. ALE presents significant research challenges for reinforcement learning, model learning, model-based planning, imitation learning, transfer learning, and intrinsic motivation. Most importantly, it provides a rigorous testbed for evaluating and comparing approaches to these problems. We illustrate the promise of ALE by presenting a benchmark set of domain-independent agents designed using well-established AI techniques for both reinforcement learning and planning. In doing so, we also propose an evaluation methodology made possible by ALE, reporting empirical results on over 55 different games. We conclude with a brief update on the latest ALE developments. All of the software, including the benchmark agents, is publicly available.
BibTeX
@inproceedings{bellemare2015arcade,
  author = {Marc G. Bellemare and Yavar Naddaf and Joel Veness and Michael Bowling},
  title = {{The Arcade Learning Environment: An Evaluation Platform for General Agents (Extended Abstract)}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
  url = {https://openreview.net/forum?id=r1EjHXGdWS},
}
Figure