Modularization of End-to-End Learning: Case Study in Arcade Games

Andrew Melnik, Sascha Fleer, Malte Schilling, Helge RitterarXiv 2019 · arXiv

TL;DR

Pure end-to-end reinforcement learning is notoriously sample-inefficient, often requiring millions of frames to master simple Atari games. This paper demonstrates that by decomposing the environment into modular, interacting components—specifically controllable and non-controllable objects—an agent can achieve human-level performance in just 10 to 15 minutes of gameplay (under 25,000 steps).

What Problem Does This Paper Address?

Deep reinforcement learning (RL) has achieved remarkable success in arcade environments, but it typically treats the game screen as a holistic, un-factored black box. Because the agent must learn everything from scratch—from basic physics to high-level strategy—it requires an enormous amount of experience (often tens of millions of frames) to achieve competence.

In contrast, humans bring rich inductive biases to a new game. We instantly recognize distinct objects, intuitively separate things we control (like a paddle) from things we don't (like a ball), and predict trajectories based on intuitive physics. The authors address this gap by moving away from pure end-to-end pixel-to-action learning, proposing instead a modular framework that exploits the underlying causal structure of the environment.

Methods

The authors introduce a framework that converts pixel-based inputs into an object-based representation (tracking shapes, types, positions, and velocities). The core architecture consists of three specialized functional modules that operate in parallel:

  1. Module 1: Non-controllable Object Module: This module uses supervised learning to predict the future trajectory of autonomous objects (e.g., the ball). A neural network processes past velocities and sensor data to recursively roll out a sequence of $n$ future positions. It includes a proximity "sensor" to account for collisions with walls or other obstacles.
  2. Module 2: Reward-Events Regression Module: This module tracks contact points between controllable and non-controllable objects (e.g., where the ball hits the paddle) and correlates them with delayed rewards. It maps out which specific interaction zones yield the highest expected payoff.
  3. Module 3: Controllable Object Module: Operating the agent's avatar (e.g., the paddle), this module treats the optimal intersection point calculated by Modules 1 and 2 as a goal position. It utilizes Hindsight Experience Replay (HER) and a learned universal-value policy to output the precise actions needed to reach that goal.

The interplay of these components is illustrated in the framework overview below:

Figure 2: A. Overview. B. Trajectory-prediction pseudo code of the non-controllable-object module. n - number of recursive prediction steps; pos - anterior position of a predicted trajectory at the prediction step i; s - observation from the sensor at the step i; v - predicted velocity vector at the step i.

Datasets

The framework was evaluated using three classic arcade environments from the OpenAI Gym Atari benchmark:

  • Breakout
  • Pong
  • Video Pinball

These environments were selected because they share generic interaction primitives (e.g., intercepting a moving projectile with a controllable tool) while differing significantly in visual appearance and game dynamics.

The three Atari environments evaluated in the study

Key Insights and Ideas

  • The Right Tool for the Job: Instead of forcing a single reinforcement learning algorithm to learn physics, value functions, and control simultaneously, the paper applies specialized paradigms where they fit best. Trajectory prediction is treated as a straightforward supervised regression problem, drastically simplifying the learning landscape.
  • Exploiting Causal Structure: Factoring the world into "things I control" and "things I don't" bypasses the classic credit assignment problem. The agent doesn't need to guess which action caused a reward minutes later; it explicitly calculates how its paddle placement altered the ball's trajectory to score points.
  • Strategic Nuance via Regression: In Pong, hitting the ball with the center of the paddle merely reflects it safely, whereas hitting it with the very edge creates a sharp, aggressive return angle that is more likely to score. Module 2 successfully captures these nuances by building a statistical reward map across the object's geometry.

Key Results

The modular framework achieved human-level control in less than 25,000 steps (equivalent to roughly 10–15 minutes of real-time play at 60 Hz).

To put this in perspective, state-of-the-art end-to-end deep RL models (such as Rainbow or standard DQN architectures) typically require 25 million steps or more to reach comparable performance. By introducing object-level modularity, the authors demonstrated a massive 1,000x reduction in sample complexity, matching the rapid learning curve of a human player.

Limitations and Future Work

The primary limitation of this study is that the object extraction process was not automated. The authors relied on hand-tuned, environment-specific pixel filters (such as color segmentation and frame-differencing) to feed the object-based data structures to the modules.

The authors note that automating this decomposition—enabling an agent to unsupervisedly discover objects and their causal relationships directly from raw pixels—remains a major open challenge. Future work will focus on expanding the library of functional modules to cover a wider variety of physical interaction primitives, allowing the system to scale to more complex, non-arcade environments.

Summary updated

Abstract

Complex environments and tasks pose a difficult problem for holistic end-to-end learning approaches. Decomposition of an environment into interacting controllable and non-controllable objects allows supervised learning for non-controllable objects and universal value function approximator learning for controllable objects. Such decomposition should lead to a shorter learning time and better generalisation capability. Here, we consider arcade-game environments as sets of interacting objects (controllable, non-controllable) and propose a set of functional modules that are specialized on mastering different types of interactions in a broad range of environments. The modules utilize regression, supervised learning, and reinforcement learning algorithms. Results of this case study in different Atari games suggest that human-level performance can be achieved by a learning agent within a human amount of game experience (10-15 minutes game time) when a proper decomposition of an environment or a task is provided. However, automatization of such decomposition remains a challenging problem. This case study shows how a model of a causal structure underlying an environment or a task can benefit learning time and generalization capability of the agent, and argues in favor of exploiting modular structure in contrast to using pure end-to-end learning approaches.
BibTeX
@misc{melnik2019modularization,
  author = {Andrew Melnik and Sascha Fleer and Malte Schilling and Helge Ritter},
  title = {{Modularization of End-to-End Learning: Case Study in Arcade Games}},
  year = {2019},
  eprint = {1901.09895},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1901.09895},
}
Figure