Convergence of Fast Policy Iteration in Markov Games and Robust MDPs

Keith Badger, Jefferson Huang, Marek Petrik

Abstract

Markov games and robust MDPs are closely related models that involve computing a pair of saddle point policies. As part of the long-standing effort to develop efficient algorithms for these models, the Filar-Tolwinski (FT) algorithm has shown considerable promise. As our first contribution, we demonstrate that FT may fail to converge to a saddle point and may loop indefinitely, even in small games. This observation contradicts the proof of FT's convergence to a saddle point in the original paper. As our second contribution, we propose Residual Conditioned Policy Iteration (RCPI). RCPI builds on FT, but is guaranteed to converge to a saddle point. Our numerical results show that RCPI outperforms other convergent algorithms by several orders of magnitude.
BibTeX
@misc{badger2025convergence,
  author = {Keith Badger and Jefferson Huang and Marek Petrik},
  title = {{Convergence of Fast Policy Iteration in Markov Games and Robust MDPs}},
  year = {2025},
  eprint = {2508.06661},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.06661},
}

Loading related content...

Figure