Learning to Play General Video-Games via an Object Embedding Network

William Woof, Ke Chen

Abstract

Deep reinforcement learning (DRL) has proven to be an effective tool for creating general video-game AI. However, most current DRL agents learn end-to-end from raw video output, which is computationally expensive and differs significantly from human perception. This paper presents a novel method enabling DRL agents to learn directly from object information via an Object Embedding Network (OEN). The OEN compresses a variable-length set of object feature vectors into a single fixed-length unified feature vector representing the game state. Experimental results on the General Video Game AI (GVG-AI) Competition framework suggest that our object-based DRL agent yields performance comparable to state-of-the-art pixel-based approaches.

Introduction: Beyond Pixel-Based Learning

Most state-of-the-art DRL agents, such as the Deep Q-Network (DQN), learn from raw RGB frames. While successful, this approach faces several challenges:

  1. Computational Waste: Processing millions of pixels is inefficient when game logic depends on only a few dozen objects.
  2. Inconsistency: Moving cameras (e.g., in StarCraft) cause pixel-level changes even when the underlying state is stable.
  3. Human Divergence: Humans perceive games as collections of entities (players, enemies, projectiles) rather than grids of color values.
  4. Generalization: Previous object-based attempts often relied on handcrafted features that failed to generalize across different game mechanics.

The OEN addresses these issues by providing a general-purpose architecture to process a variable number of game objects directly.

Methods: The Object Embedding Network (OEN)

The core innovation is an architecture based on "set networks." Unlike standard neural networks requiring fixed-size inputs, the OEN handles a dynamic list of objects that changes as entities appear or disappear.

Figure 2: The process of converting a game screen into a list of object classes and coordinates, which are then transformed into feature vectors.

The OEN operates in three distinct stages:

  1. Embedding: Each object (represented by coordinates, class, etc.) passes through an identical embedding network, mapping raw data into a higher-dimensional space.
  2. Global Pooling: To manage the variable number of objects, an element-wise max-pooling layer squashes individual embeddings into a single, fixed-length "unified feature vector."
  3. Task Network: This vector is fed into a fully-connected network to predict Q-values for possible actions.

This architecture ensures permutation invariance, meaning the arbitrary order of objects in the game engine's list does not affect the agent's decision-making.

Evaluation and Datasets

The researchers evaluated the OEN agent using the General Video Game AI (GVG-AI) Competition framework. They selected five diverse arcade-style games to test generalization capabilities.

Figure 5: The five GVG-AI games used for evaluation: (a) Aliens, (b) Boulderdash, (c) Missile Command, (d) Survive Zombies, and (e) Zelda.

The OEN agent was compared against two baselines: an Image-based agent (standard CNN) and a Feature-based agent (using handcrafted distance features).

Key Results and Insights

The OEN-based agent proved to be a highly effective alternative to traditional pixel-based DRL:

  • Competitive Performance: In all five games, the OEN agent performed comparably to or better than the CNN and handcrafted baselines.
  • Superiority in Specific Titles: The OEN agent achieved the highest mean scores in Aliens, Missile Command, and Zelda. In Missile Command, it scored 10.44, nearly doubling the Image-based agent's score of 5.72.
  • Efficiency and Abstraction: By focusing on objects, the agent ignores superfluous visual data (like background textures), allowing it to learn the core logic of the environment more effectively.

Conclusion and Future Work

While the OEN demonstrates strong performance, it currently assumes direct access to the game's internal object list. Future work could integrate computer vision steps, such as semantic segmentation, to extract objects from pixels automatically. Additionally, incorporating more sophisticated relational modules could enhance the agent's ability to understand complex interactions between specific pairs of objects, further closing the gap between AI and human-like game perception.

Summary updated

Abstract

Deep reinforcement learning (DRL) has proven to be an effective tool for creating general video-game AI. However most current DRL video-game agents learn end-to-end from the video-output of the game, which is superfluous for many applications and creates a number of additional problems. More importantly, directly working on pixel-based raw video data is substantially distinct from what a human player does.In this paper, we present a novel method which enables DRL agents to learn directly from object information. This is obtained via use of an object embedding network (OEN) that compresses a set of object feature vectors of different lengths into a single fixed-length unified feature vector representing the current game-state and fulfills the DRL simultaneously. We evaluate our OEN-based DRL agent by comparing to several state-of-the-art approaches on a selection of games from the GVG-AI Competition. Experimental results suggest that our object-based DRL agent yields performance comparable to that of those approaches used in our comparative study.
BibTeX
@inproceedings{woof2018learning,
  author = {William Woof and Ke Chen},
  title = {{Learning to Play General Video-Games via an Object Embedding Network}},
  booktitle = {Cig},
  year = {2018},
  doi = {10.1109/cig.2018.8490438},
}
Figure