Introduction
Traditional "superhuman" game AIs, such as the original AlphaZero, are highly specialized. They rely on handcrafted neural network architectures tailored to specific board geometries (e.g., the 19x19 grid of Go) and are typically restricted to two-player, zero-sum, turn-based games. General Game Playing (GGP) presents a significantly greater challenge: agents must learn to play any game described at runtime via the Game Description Language (GDL), without prior knowledge of rules, goals, or player counts.
This paper introduces "Generalised AlphaZero," an extension of the AlphaZero algorithm designed for the GGP domain. By replacing game-specific architectures with general propositional networks and multi-head outputs, the authors created an agent capable of mastering diverse game types, including multi-player and cooperative scenarios.
Architecture: From Boards to Logic Graphs
To achieve generality, the authors moved away from Convolutional Neural Networks (CNNs), which assume spatial locality. Instead, they utilized Propositional Networks to convert GDL logic rules into a graph of boolean nodes. The state of these nodes is then processed by a standard Multi-Layer Perceptron (MLP).
As shown in the architecture diagram, the network uses a shared "feature extraction" base that branches into separate "heads" for each player. This allows the agent to handle asymmetric games (where players have different roles) and multi-player games by learning general game features while maintaining role-specific policy and value predictions.
Training Strategy and Hybrid Value Targets
A key innovation in this framework is the use of Hybrid Value Targets. Standard AlphaZero trains its value network based solely on the final game result ($z$). In the complex and often sparse reward environments of GGP, the authors found that a weighted average of the final result and the Monte Carlo Tree Search (MCTS) search value ($q$) provides a more stable learning signal.
Empirical results demonstrate that a weight of $r=0.5$ (mixing the final outcome with the MCTS estimate) is the "sweet spot" for learning. This balance helps the agent ground its long-term strategy in truth while maintaining short-term search consistency. Additionally, the authors implemented generalized exploration by scaling Dirichlet noise inversely with the number of legal moves, ensuring robust discovery across different game states.
Experimental Evaluation
The agent was evaluated on four distinct games from past GGP competitions:
- Connect-4 (6x7): A standard 2-player, zero-sum benchmark.
- Breakthrough (6x6): A 2-player game with a high branching factor.
- Babel: A 3-player cooperative game requiring tower construction.
- Pacman 3p: A 3-player asymmetric game (1 Pacman vs. 2 Ghosts) with simultaneous moves.
Key Results
- Breakthrough Dominance: Generalised AlphaZero achieved a near 100% win rate against the UCT benchmark after only 10 hours of training, effectively navigating the game's high branching factor.
- Asymmetric Mastery: In Pacman, the agent learned coordinated strategies for the ghosts to trap Pacman, significantly outperforming UCT.
- Cooperation Hurdles: In the cooperative game Babel, the agent improved over time but struggled to consistently beat a basic UCT baseline. The authors suggest that DRL agents may require more explicit coordination mechanisms to master complex cooperative play.
Conclusion and Future Work
While Generalised AlphaZero demonstrates remarkable versatility, its primary limitation remains computational cost. Training for a specific game can take up to 30 hours, which is currently impractical for real-time competitive GGP tournaments. Future research aims to incorporate symmetry detection and more advanced feature extraction to combine the spatial reasoning of CNNs with the flexibility of the current MLP-based approach.
Summary updated