Abstract
Deep reinforcement learning (DRL) has proven to be an effective tool for creating general video-game AI. However, most current DRL agents learn end-to-end from raw video output, which is computationally expensive and differs significantly from human perception. This paper presents a novel method enabling DRL agents to learn directly from object information via an Object Embedding Network (OEN). The OEN compresses a variable-length set of object feature vectors into a single fixed-length unified feature vector representing the game state. Experimental results on the General Video Game AI (GVG-AI) Competition framework suggest that our object-based DRL agent yields performance comparable to state-of-the-art pixel-based approaches.
Introduction: Beyond Pixel-Based Learning
Most state-of-the-art DRL agents, such as the Deep Q-Network (DQN), learn from raw RGB frames. While successful, this approach faces several challenges:
- Computational Waste: Processing millions of pixels is inefficient when game logic depends on only a few dozen objects.
- Inconsistency: Moving cameras (e.g., in StarCraft) cause pixel-level changes even when the underlying state is stable.
- Human Divergence: Humans perceive games as collections of entities (players, enemies, projectiles) rather than grids of color values.
- Generalization: Previous object-based attempts often relied on handcrafted features that failed to generalize across different game mechanics.
The OEN addresses these issues by providing a general-purpose architecture to process a variable number of game objects directly.
Methods: The Object Embedding Network (OEN)
The core innovation is an architecture based on "set networks." Unlike standard neural networks requiring fixed-size inputs, the OEN handles a dynamic list of objects that changes as entities appear or disappear.
The OEN operates in three distinct stages:
- Embedding: Each object (represented by coordinates, class, etc.) passes through an identical embedding network, mapping raw data into a higher-dimensional space.
- Global Pooling: To manage the variable number of objects, an element-wise max-pooling layer squashes individual embeddings into a single, fixed-length "unified feature vector."
- Task Network: This vector is fed into a fully-connected network to predict Q-values for possible actions.
This architecture ensures permutation invariance, meaning the arbitrary order of objects in the game engine's list does not affect the agent's decision-making.
Evaluation and Datasets
The researchers evaluated the OEN agent using the General Video Game AI (GVG-AI) Competition framework. They selected five diverse arcade-style games to test generalization capabilities.
The OEN agent was compared against two baselines: an Image-based agent (standard CNN) and a Feature-based agent (using handcrafted distance features).
Key Results and Insights
The OEN-based agent proved to be a highly effective alternative to traditional pixel-based DRL:
- Competitive Performance: In all five games, the OEN agent performed comparably to or better than the CNN and handcrafted baselines.
- Superiority in Specific Titles: The OEN agent achieved the highest mean scores in Aliens, Missile Command, and Zelda. In Missile Command, it scored 10.44, nearly doubling the Image-based agent's score of 5.72.
- Efficiency and Abstraction: By focusing on objects, the agent ignores superfluous visual data (like background textures), allowing it to learn the core logic of the environment more effectively.
Conclusion and Future Work
While the OEN demonstrates strong performance, it currently assumes direct access to the game's internal object list. Future work could integrate computer vision steps, such as semantic segmentation, to extract objects from pixels automatically. Additionally, incorporating more sophisticated relational modules could enhance the agent's ability to understand complex interactions between specific pairs of objects, further closing the gap between AI and human-like game perception.
Summary updated