Adversarial Policies Beat Professional-Level Go AIs

Tony Tong Wang, Adam Gleave, Nora Belrose, Tom Tseng, Joseph Miller, Michael D. Dennis, Yawen Duan, Viktor Pogrebniak, Sergey Levine, Stuart Russell

TL;DR

Researchers have demonstrated that even superhuman AI systems like KataGo can be consistently defeated by adversarial policies that exploit specific blind spots rather than playing "better" Go. These attacks are so effective and comprehensible that human amateurs can learn the strategy to beat professional-level bots, highlighting a critical lack of worst-case robustness in state-of-the-art reinforcement learning.

What Problem Does This Paper Address?

In recent years, reinforcement learning (RL) has produced agents with staggering average-case performance in complex games like Go. However, a major open question remains: does superhuman capability imply robustness? In other words, does being "the best" at a game mean you have no catastrophic failure modes?

This paper investigates the worst-case robustness of state-of-the-art Go AIs. The authors set out to discover if an adversary—one that might be a poor Go player overall—can systematically trick a superhuman agent into making game-losing blunders. They chose KataGo as their primary target because it is the strongest publicly available Go AI and was expected to be the most challenging to exploit.

Methods

The authors employ a "victim-play" training regime, a twist on the standard AlphaZero-style self-play. Instead of playing against itself, an adversary is trained against a fixed "victim" agent.

Key methodological innovations include:

  • Adversarial MCTS (A-MCTS): In standard MCTS, an agent models its opponent as playing like itself. In A-MCTS, the adversary explicitly models the victim's behavior by sampling from the victim's own policy network during its tree search. This allows the adversary to "anticipate" the victim's specific reactions to its moves.
  • Curriculum Learning: To overcome the difficulty of a randomly initialized network learning to beat a superhuman opponent, the authors used a curriculum. They started by attacking early, weak versions of KataGo and gradually increased the victim's strength and search budget as the adversary improved.
  • Pass-Alive Defense Hardening: After finding an initial "pass-trick" exploit, the authors added a hard-coded defense to the victim to prevent premature passing. This forced the discovery of the more sophisticated "cyclic" attack.
Figure 4.1. MCTS (left) builds a search tree one node at a time. To add a node, it walks down the tree until a new leaf is reached (red arrows). At a node x, the next step of the walk is determined by a PUCT (Rosin, 2011) algorithm (solid arrows) which takes into account neural network evaluations of each node in the sub- tree of x. A-MCTS-S (middle) walks down the tree by using a modified PUCT algorithm at adversary nodes, and sampling di- rectly from the victim’s policy network (dashed arrows) at victim nodes. A-MCTS-R (right) performs a full simulation of the vic- tim as opposed to sampling from the victim’s policy net. Search trees are depicted as binary for illustrative purposes only. See Ap- pendix B for full details.

Datasets

The study doesn't rely on a static dataset but rather on the KataGo training history, utilizing various checkpoints released by the KataGo team. The training data for the adversary consists of millions of games generated through the victim-play process. For evaluation and transfer tests, the authors also used other superhuman Go AIs: ELF OpenGo and Leela Zero.

Key Insights and Ideas

  • Non-Transitivity of Skill: The most striking insight is the breakdown of the traditional skill hierarchy. The adversary beats KataGo, KataGo beats human professionals, and professionals beat amateurs. However, a human amateur can easily beat the adversary. This proves the adversary is not a "strong" Go player in a general sense; it is a specialist at exploiting a specific algorithmic blind spot. The "BEATS?!" cycle showing non-transitivity
  • The Cyclic Vulnerability: The primary exploit involves luring the victim into creating a large circular group of stones. The victim's neural network appears to fundamentally misjudge the safety of these "cyclic" topologies, likely "double-counting" liberties and believing the group is safe when it is actually one move away from capture.
  • Search is Not a Silver Bullet: While increasing the victim's search budget (MCTS visits) improves its robustness, it does not solve the problem. The adversary still maintains a high win rate even when the victim performs massive amounts of search (up to 10 million nodes per move), which is far beyond what is used in professional tournament play.
  • Human-Implementable Attacks: Unlike adversarial "noise" in computer vision, this attack is high-level and strategic. A human Go expert was able to learn the cyclic strategy from the adversary's game records and successfully beat top-tier bots manually, without any computer assistance.
Figure 2.1. A human amateur beats our adversarial policy (Appendix J.1) that beats KataGo. This non-transitivity shows the adversary is not a generally capable policy, and is just exploiting KataGo.

Key Results

  • Superhuman Defeat: The adversarial policy achieved a >97% win rate against KataGo at its strongest superhuman settings.
  • Compute Efficiency: The adversary was trained using less than 14% of the compute budget required to train the original KataGo victim.
  • Zero-Shot Transfer: The attack transferred "zero-shot" to other superhuman AIs, winning 6.1% of games against Leela Zero and 3.5% against ELF OpenGo, despite never having seen them during training.
  • Defense Persistence: Even when KataGo was adversarially trained to defend against this specific cyclic attack, the authors were able to fine-tune their adversary to break the new defense, suggesting the vulnerability is deep-seated.
Figure 5.1. The win rate (y-axis) of the cyclic-adversary over time (x-axis) playing with 600 visits against four different victims. The strongest cyclic-adversary checkpoint (marked ♦) wins 1048/1048 = 100% games against Latestdef without search and 1007/1052 = 95.7% games against Latestdef with 4096 visits. The shaded interval is a 95% Clopper-Pearson interval over 50 games per checkpoint. The cyclic-adversary is trained with a curriculum, starting from cp39def without search and ending at Latestdef with 131,072 visits. Vertical dotted lines denote switches to stronger victim networks or to an increase in Latestdef’s search budget.

Limitations and Future Work

The authors note that Go's specific rules and the topological nature of the "cyclic" exploit might make it uniquely vulnerable compared to other domains. Furthermore, while search helps, it is computationally expensive and doesn't provide a complete defense.

Promising future directions include:

  • Evaluating these attacks in other complex domains like language modeling or autonomous driving.
  • Developing more robust training methods, such as population-based training or incorporating game-theoretic approaches like Counterfactual Regret Minimization (CFR) that are designed to be unexploitable.
  • Mechanistic interpretability work to understand exactly why the neural network's internal activations fail on cyclic groups.

Summary updated

Abstract

We attack the state-of-the-art Go-playing AI system KataGo by training adversarial policies against it, achieving a >97% win rate against KataGo running at superhuman settings. Our adversaries do not win by playing Go well. Instead, they trick KataGo into making serious blunders. Our attack transfers zero-shot to other superhuman Go-playing AIs, and is comprehensible to the extent that human experts can implement it without algorithmic assistance to consistently beat superhuman AIs. The core vulnerability uncovered by our attack persists even in KataGo agents adversarially trained to defend against our attack. Our results demonstrate that even superhuman AI systems may harbor surprising failure modes. Example games are available https://goattack.far.ai/.
BibTeX
@inproceedings{wang2023adversarial,
  author = {Tony Tong Wang and Adam Gleave and Nora Belrose and Tom Tseng and Joseph Miller and Michael D. Dennis and Yawen Duan and Viktor Pogrebniak and Sergey Levine and Stuart Russell},
  title = {{Adversarial Policies Beat Professional-Level Go AIs}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=LX3VAhXNTw},
}
Figure