R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training

Gengsheng Li, Jinghan He, Shijie Wang, Ruiqi Liu, Renrui Zhang, Zijun Yao, Junfeng Fang, Haiyun Guo, Dan Zhang, Jinqiao WangICML 2026 · OpenReview

Abstract

Self-play bootstraps LLM reasoning through an iterative Challenger--Solver loop: the Challenger is trained to generate questions that target the Solver's capabilities, and the Solver is optimized on the generated data to expand its reasoning skills. However, existing frameworks like R-Zero often exhibit non-sustained improvement, where early gains degrade as self-play continues. We identify a key failure mode, **Diversity Illusion**, where the Solver's training signals appear diverse yet collapse into recurring underlying patterns. It manifests as (1) *Local Diversity Illusion*, where diversity is enforced only within-batch, inducing cross-iteration mode cycling; and (2) *Surface Diversity Illusion*, where questions vary superficially but require near-identical reasoning skills. To mitigate them, we propose R-Diverse with two aligned innovations: **Memory-Augmented Penalty (MAP)**, which uses a persistent memory bank to discourage recycling across iterations, and **Skill-Aware Measurement (SAM)**, which evaluates diversity by the reasoning skills exercised rather than surface variation of questions. Across 10 math and general reasoning benchmarks, R-Diverse sustains gains over more iterations and consistently outperforms prior self-play methods.
BibTeX
@inproceedings{li2026r,
  author = {Gengsheng Li and Jinghan He and Shijie Wang and Ruiqi Liu and Renrui Zhang and Zijun Yao and Junfeng Fang and Haiyun Guo and Dan Zhang and Jinqiao Wang},
  title = {{R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=DZiuKVvrJW},
}

Loading related content...

Figure