CasePlay: Self-Play Reinforcement Learning from Case Reports for Medical Reasoning

Hao Wang, Zihan Wang, Kai Wu, Jian Yao, Yiwen Ye, Yongcan Luo, Dapeng Wu, Jiale Chen, Haihua YangarXiv 2026 · arXiv

TL;DR

CasePlay is a self-play reinforcement learning (RL) framework that transforms unlabeled clinical case reports into high-quality training signals for medical LLMs. By conditioning question generation on specific "knowledge points" and using a multi-dimensional rubric for feedback, it overcomes the common pitfalls of repetitive questioning and shallow reasoning, leading to significant gains across medical benchmarks.

What Problem Does This Paper Address?

Improving the medical reasoning of Large Language Models (LLMs) typically requires massive amounts of expert-labeled data or expensive synthetic pipelines. While the medical literature is full of high-quality, unlabeled clinical case reports, turning these documents into effective training data is surprisingly difficult.

The authors identify two "pitfalls" in standard document-grounded self-play:

  1. Knowledge-Point Fixation: A model tasked with generating questions from a report tends to "anchor" on the most obvious clues (like the final diagnosis) and ignores the subtle, information-dense journey of the clinical course.
  2. Difficulty-Quality Mismatch: Standard RL often rewards "intermediate difficulty" questions (where the model gets it right ~50% of the time). However, in medicine, a question can be "hard" simply because it is poorly phrased or relies on trivia, rather than requiring deep clinical integration.

Methods

CasePlay introduces a co-evolutionary loop between a Proposer (which generates questions) and a Reasoner (which solves them).

Figure 1: Overview of CasePlay.

The framework features two novel components:

  • Knowledge-Conditioned Proposer: Before training, the system extracts multiple "knowledge points" (KPs) from each case report (e.g., specific symptom progressions or treatment responses). During self-play, the proposer is forced to generate questions based on a specific KP using a round-robin strategy. This ensures the model explores the entire document rather than just the highlights.
  • Rubric-Judge Reasoner: Instead of providing a simple binary reward (correct/incorrect), the reasoner evaluates the proposer’s questions against a rubric:
  • Clue Integration: Does the question require synthesizing multiple clinical clues?
  • Reasoning Depth: Does it require multi-hop clinical inference?
  • Option Quality: Are the distractors plausible and discriminative?

The models are optimized using Group Relative Policy Optimization (GRPO) in an alternating fashion: the proposer is updated to create better questions, and the reasoner is updated to solve them more accurately.

Datasets

  • Training: A corpus of 4,281 clinical case reports filtered from PubMed Central (PMC) published between 2020 and 2025.
  • Evaluation: The authors test on seven benchmarks across three categories:
  • Expert Medical Reasoning: MedXpert-Text, MedBullets (OP4/OP5), and MedQA-USMLE.
  • Clinical Case Reasoning: LiveClin, which focuses on patient narratives.
  • Scientific QA: MMLU-Pro (Health/Biology) and GPQA (Genetics/Molecular Biology).

Key Insights and Ideas

  • Forced Exploration is Necessary: Without explicit conditioning, document coverage saturates quickly. As shown in Figure 2(a), standard methods (like SPICE) stop finding new information in reports, while CasePlay continues to increase its "document coverage ratio" by cycling through different knowledge points.
  • Difficulty $\neq$ Quality: The authors found that 31.3% of "intermediate difficulty" questions generated by baselines were actually low quality (rubric score < 0.6). By rewarding the proposer for quality (clue integration and depth) alongside learnability (difficulty), CasePlay generates much more effective training examples.
  • Diversity in Question Types: Conditioning on KPs naturally leads to a more balanced distribution of question types (diagnosis, treatment, mechanism, etc.), whereas unconditioned models tend to obsess over diagnosis-related questions.
Figure 2: (a) Cumulative source-document coverage of generated questions over training. (b) MDS [26] projection of generated-question embeddings. (c) Rubric-based quality-score distribution among intermediate-difficulty questions with reasoner accuracy of 0.5.

Key Results

CasePlay consistently outperformed both medical reasoning models and existing self-play baselines:

  • Average Gain: Improved the warm-start (SFT) model by 3.1 points on average across all benchmarks.
  • LiveClin Performance: Achieved a 6.3-point gain on the LiveClin benchmark, which is particularly significant as LiveClin mimics real-world clinical narratives.
  • Robustness: The method proved effective across different model scales (4B to 8B parameters) and different model families (Qwen3, OctoThinker).
  • Ablation: Removing the "knowledge-point" conditioning or the "rubric-judge" feedback led to significant performance drops, confirming that both components are vital.

Limitations and Future Work

  • Static Knowledge Points: Currently, KPs are extracted once before RL begins. The authors suggest that "online discovery" or refining KPs as the model improves could be a promising next step.
  • Text-Only Focus: The current framework ignores images (X-rays, pathology slides) often found in case reports. Extending CasePlay to a multimodal setting is a clear direction for future research.
  • Case Report Bias: Case reports often highlight "unusual" or "publication-worthy" cases rather than routine presentations, which might introduce specific biases into the trained model.

Summary updated

Abstract

Reinforcement learning (RL) is a promising way to improve the clinical reasoning abilities of large language models (LLMs), but current pipelines often rely on expert-labeled questions or costly synthetic-data generation. Meanwhile, the medical literature offers a wealth of high-quality unlabeled documents, including clinical case reports that trace the full clinical course from initial presentation to final treatment decisions. We study whether document-grounded self-play can convert these reports into effective RL training signals. We identify two challenges that limit existing self-play methods in this setting: knowledge-point fixation}, where the proposer repeatedly asks about a narrow set of salient clues in a source report, and difficulty-quality mismatch, where appropriately difficult questions may still be poorly formed or shallow, limiting their value for reasoner training. To address these challenges, we propose CasePlay, a self-play framework with two key components: (a) a Knowledge-Conditioned Proposer, which anchors question generation to case-specific knowledge points so that questions draw on a broader range of evidence from the source report; and (b) a Rubric-Judge Reasoner, which augments answer-accuracy feedback with rubric-based quality feedback on clue integration, reasoning depth, and option quality, guiding the proposer toward reasoning-intensive questions that better support reasoner training. Extensive experiments across seven medically related benchmarks demonstrate that CasePlay outperforms existing self-play baselines and improves the SFT warm-start model by 3.1 points on average, highlighted by a 6.3-point gain on LiveClin. This approach offers an effective path toward continually improving medical reasoning ability through self-play RL.
BibTeX
@misc{wang2026caseplay,
  author = {Hao Wang and Zihan Wang and Kai Wu and Jian Yao and Yiwen Ye and Yongcan Luo and Dapeng Wu and Jiale Chen and Haihua Yang},
  title = {{CasePlay: Self-Play Reinforcement Learning from Case Reports for Medical Reasoning}},
  year = {2026},
  eprint = {2610.24952},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.24952},
}
Figure