TL;DR
Researchers have demonstrated that even superhuman AI systems like KataGo can be consistently defeated by adversarial policies that exploit specific blind spots rather than playing "better" Go. These attacks are so effective and comprehensible that human amateurs can learn the strategy to beat professional-level bots, highlighting a critical lack of worst-case robustness in state-of-the-art reinforcement learning.
What Problem Does This Paper Address?
In recent years, reinforcement learning (RL) has produced agents with staggering average-case performance in complex games like Go. However, a major open question remains: does superhuman capability imply robustness? In other words, does being "the best" at a game mean you have no catastrophic failure modes?
This paper investigates the worst-case robustness of state-of-the-art Go AIs. The authors set out to discover if an adversary—one that might be a poor Go player overall—can systematically trick a superhuman agent into making game-losing blunders. They chose KataGo as their primary target because it is the strongest publicly available Go AI and was expected to be the most challenging to exploit.
Methods
The authors employ a "victim-play" training regime, a twist on the standard AlphaZero-style self-play. Instead of playing against itself, an adversary is trained against a fixed "victim" agent.
Key methodological innovations include:
- Adversarial MCTS (A-MCTS): In standard MCTS, an agent models its opponent as playing like itself. In A-MCTS, the adversary explicitly models the victim's behavior by sampling from the victim's own policy network during its tree search. This allows the adversary to "anticipate" the victim's specific reactions to its moves.
- Curriculum Learning: To overcome the difficulty of a randomly initialized network learning to beat a superhuman opponent, the authors used a curriculum. They started by attacking early, weak versions of KataGo and gradually increased the victim's strength and search budget as the adversary improved.
- Pass-Alive Defense Hardening: After finding an initial "pass-trick" exploit, the authors added a hard-coded defense to the victim to prevent premature passing. This forced the discovery of the more sophisticated "cyclic" attack.
Datasets
The study doesn't rely on a static dataset but rather on the KataGo training history, utilizing various checkpoints released by the KataGo team. The training data for the adversary consists of millions of games generated through the victim-play process. For evaluation and transfer tests, the authors also used other superhuman Go AIs: ELF OpenGo and Leela Zero.
Key Insights and Ideas
- Non-Transitivity of Skill: The most striking insight is the breakdown of the traditional skill hierarchy. The adversary beats KataGo, KataGo beats human professionals, and professionals beat amateurs. However, a human amateur can easily beat the adversary. This proves the adversary is not a "strong" Go player in a general sense; it is a specialist at exploiting a specific algorithmic blind spot. The "BEATS?!" cycle showing non-transitivity
- The Cyclic Vulnerability: The primary exploit involves luring the victim into creating a large circular group of stones. The victim's neural network appears to fundamentally misjudge the safety of these "cyclic" topologies, likely "double-counting" liberties and believing the group is safe when it is actually one move away from capture.
- Search is Not a Silver Bullet: While increasing the victim's search budget (MCTS visits) improves its robustness, it does not solve the problem. The adversary still maintains a high win rate even when the victim performs massive amounts of search (up to 10 million nodes per move), which is far beyond what is used in professional tournament play.
- Human-Implementable Attacks: Unlike adversarial "noise" in computer vision, this attack is high-level and strategic. A human Go expert was able to learn the cyclic strategy from the adversary's game records and successfully beat top-tier bots manually, without any computer assistance.
Key Results
- Superhuman Defeat: The adversarial policy achieved a >97% win rate against KataGo at its strongest superhuman settings.
- Compute Efficiency: The adversary was trained using less than 14% of the compute budget required to train the original KataGo victim.
- Zero-Shot Transfer: The attack transferred "zero-shot" to other superhuman AIs, winning 6.1% of games against Leela Zero and 3.5% against ELF OpenGo, despite never having seen them during training.
- Defense Persistence: Even when KataGo was adversarially trained to defend against this specific cyclic attack, the authors were able to fine-tune their adversary to break the new defense, suggesting the vulnerability is deep-seated.
Limitations and Future Work
The authors note that Go's specific rules and the topological nature of the "cyclic" exploit might make it uniquely vulnerable compared to other domains. Furthermore, while search helps, it is computationally expensive and doesn't provide a complete defense.
Promising future directions include:
- Evaluating these attacks in other complex domains like language modeling or autonomous driving.
- Developing more robust training methods, such as population-based training or incorporating game-theoretic approaches like Counterfactual Regret Minimization (CFR) that are designed to be unexploitable.
- Mechanistic interpretability work to understand exactly why the neural network's internal activations fail on cyclic groups.
Summary updated