Learning to Play General Video-Games via an Object Embedding Network

Ke Chen, William Woof, Ke Chen 0001

TL;DR

This paper introduces the Object Embedding Network (OEN), a novel Deep Reinforcement Learning (DRL) architecture that allows AI agents to learn from game objects rather than raw pixels. By treating a game state as a "set" of objects with varying attributes, the OEN achieves performance comparable to—and sometimes better than—traditional pixel-based agents while using a more human-like representation of the game world.

What Problem Does This Paper Address?

Most state-of-the-art DRL agents (like the famous Deep Q-Network) learn "end-to-end" from raw video frames. While impressive, this approach has several drawbacks:

  1. Computational Waste: Processing millions of pixels is expensive, especially when the underlying game logic only involves a few dozen objects.
  2. Inconsistency: In games with moving cameras (like StarCraft), the pixel view changes constantly even if the game state remains stable.
  3. Human Divergence: Humans don't perceive games as a grid of RGB values; we see "the player," "the enemy," and "the projectile."
  4. Handcrafting Trap: Previous attempts to use object data often relied on handcrafted features (e.g., "distance to nearest enemy"), which are labor-intensive to design and don't generalize well across different games.

The authors aim to bridge this gap by creating a general-purpose way to feed a variable number of game objects into a neural network.

Methods

The core innovation is the Object Embedding Network (OEN), which is based on the concept of "set networks." Unlike standard neural networks that require a fixed-size input, the OEN can handle a list of objects that changes in length as things appear or disappear on screen.

The process works in three stages:

  1. Embedding: Each object (represented by a feature vector of its coordinates, class, etc.) is passed through an identical "embedding network." This maps raw object data into a higher-dimensional space.
  2. Global Pooling: To handle the variable number of objects, the network uses an element-wise max-pooling layer. This "squashes" all the individual object embeddings into a single, fixed-length "unified feature vector" that represents the entire game state.
  3. Task Network: This unified vector is fed into a standard fully-connected network that predicts Q-values (the expected reward for each possible action).

The authors also incorporate a "contextual" embedding using an equivariant transformation. This allows the network to consider an object's relationship to the rest of the set (e.g., "how far is this alien from the average position of all aliens?").

Datasets

The researchers evaluated their agent using the General Video Game AI (GVG-AI) Competition framework. They selected five classic-style arcade games from Test Set 1:

  • Aliens (Space Invaders clone)
  • Boulderdash
  • Missile Command
  • Survive Zombies
  • Zelda (Dungeon crawling)

They compared the OEN agent against two baselines: an Image-based agent (using a standard CNN) and a Feature-based agent (using a fixed set of handcrafted distance-based features).

Key Insights and Ideas

  • Permutation Invariance: Because the OEN uses max-pooling, the order in which objects are listed doesn't matter. This is crucial because the "list of objects" provided by a game engine is often arbitrary.
  • Efficiency through Abstraction: By working with objects, the agent ignores "superfluous" visual data (like background textures or decorative sprites) that don't affect gameplay, allowing it to focus on the logic of the environment.
  • Generalization: Unlike handcrafted features, the OEN learns which object attributes are important for a specific game automatically during the training process.

Key Results

The OEN-based agent proved to be a highly effective alternative to pixel-based DRL:

  • Competitive Performance: In all five games, the OEN agent performed comparably to the CNN and handcrafted baselines.
  • Superiority in Specific Titles: The OEN agent achieved the highest mean scores in Aliens, Missile Command, and Zelda. For instance, in Missile Command, the OEN scored 10.44, significantly outperforming the Image-based agent's 5.72.
  • Robustness: While the handcrafted "features" baseline performed well in Boulderdash, it struggled in Aliens, highlighting the OEN's superior ability to generalize across different game mechanics.

Limitations and Future Work

  • Requirement for Ground Truth: The current model assumes the agent has direct access to the game's internal object list. In real-world scenarios or games where this isn't available, an additional computer vision step (like semantic segmentation) would be needed to extract objects from pixels first.
  • Fixed Attribute Sets: The OEN currently requires a fixed number of attributes per object (e.g., every object must have X and Y coordinates). The authors suggest using class-specific embedding functions in the future to handle more complex, heterogeneous object types.
  • Relational Reasoning: While the "equivariant" transformation helps, the authors note that more sophisticated relational modules (like those used in Relational Networks) could further improve the agent's ability to understand complex interactions between specific pairs of objects.

Summary updated

Abstract

Deep reinforcement learning (DRL) has proven to be an effective tool for creating general video-game AI. However most current DRL video-game agents learn end-to-end from the video-output of the game, which is superfluous for many applications and creates a number of additional problems. More importantly, directly working on pixel-based raw video data is substantially distinct from what a human player does.In this paper, we present a novel method which enables DRL agents to learn directly from object information. This is obtained via use of an object embedding network (OEN) that compresses a set of object feature vectors of different lengths into a single fixed-length unified feature vector representing the current game-state and fulfills the DRL simultaneously. We evaluate our OEN-based DRL agent by comparing to several state-of-the-art approaches on a selection of games from the GVG-AI Competition. Experimental results suggest that our object-based DRL agent yields performance comparable to that of those approaches used in our comparative study.
BibTeX
@inproceedings{chen2018learning,
  author = {Ke Chen and William Woof and Ke Chen},
  title = {{Learning to Play General Video-Games via an Object Embedding Network}},
  booktitle = {Cig},
  year = {2018},
  doi = {10.1109/cig.2018.8490438},
}

Loading related content...

Figure